Display systems and methods for determining vertical alignment between left and right displays and a user&#39;s eyes

ABSTRACT

A wearable device may include a head-mounted display (HMD) for rendering a three-dimensional (3D) virtual object which appears to be located in an ambient environment of a user of the display. The relative positions of the HMD and one or more eyes of the user may not be in desired positions to receive image information outputted by the HMD. For example, the HAMID-to-eye vertical alignment may be different between the left and right eyes. The wearable device may determine if the HMD is level on the user&#39;s head and may then provide the user with a left-eye alignment marker and a right-eye alignment marker. Based on user feedback, the wearable device may determine if there is any left-right vertical misalignment and may take actions to reduce or minimize the effects of any misalignment.

PRIORITY CLAIM

This application is a continuation of U.S. patent application Ser. No.17/890,998, filed Aug. 18, 2022, which is a continuation of U.S. patentapplication Ser. No. 17/258,083, filed on Jan. 5, 2021, now U.S. Pat.No. 11,422,620, which is a 371 of International Application No.PCT/US2019/043302, filed Jul. 24, 2019, which claims benefit of U.S.Provisional Application No. 62/702,865, filed Jul. 24, 2018.

INCORPORATION BY REFERENCE

This application incorporates by reference the entirety of each of thefollowing patent applications: U.S. Provisional Application No.62/714,649, filed on Aug. 3, 2018; U.S. Provisional Application No.62/875,474, filed on Jul. 17, 2019; U.S. application Ser. No.16/251,017, filed on Jul. 17, 2019; and PCT Application No.PCT/US2019/043096, filed on Jul. 23, 2019.

FIELD

The present disclosure relates to display systems, including virtualreality and augmented reality display systems, and, more particularly,to systems and methods for aligning left and right displays in awearable display system.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Various examples of display alignment in a mixed reality system aredisclosed.

In some embodiments, an augment reality system is provided. Theaugmented reality system comprises a head-mounted display configured topresent virtual content by outputting light to a user, an imaging deviceconfigured to capture images of eyes of the user, and at least oneprocessor communicatively coupled to the head-mounted display and theimaging device. The at least one processor is configured to provide,with the left-eye display, a left-eye alignment marker; provide, withthe right-eye display, a right-eye alignment marker; receive user inputto adjust at least one of the left-eye and the right-eye alignmentmarkers; and vertically adjust image content in the at least one of theleft-eye and right-eye displays based on the received user input. Insome embodiments, the at least one processor is configured to determineif the head-mounted display system is level relative to the user's leftand right eyes; and to provide the user with feedback on whether thehead-mounted display system is level relative to the user's left andright eyes before providing the left or right-eye alignment marker.

In some other embodiments, a method is provided for vertically aligning,with a user's left and right eyes, displayed content of a left-eyedisplay and a right-eye display of a head-mounted display system. Themethod includes providing, with the left-eye display, a left-eyealignment marker; providing, with the right-eye display, a right-eyealignment marker; receiving user input to adjust at least one of theleft-eye and the right-eye alignment markers; and vertically adjustingimage content in at least one of the left-eye and right-eye displaysbased on the received user input. In some embodiments, the methodfurther comprises determining a level of the head-mounted display systemrelative to the user's left and right eyes; providing the user feedbackon whether the head-mounted display system is level relative to theuser's left and right eyes before providing the left or right-eyealignment marker.

In some embodiments, an augment reality system is provided. Theaugmented reality system comprises a head-mounted display (TIMID)configured to present virtual content by outputting light to a user andat least one processor communicatively coupled to the HMD. The HMDcomprises a left-eye display configured to present virtual content tothe user's left eye and a right-eye display configured to presentvirtual content to the user's right eye. The at least one processor isconfigured to provide, with the left-eye display, a left-eye alignmentmarker; provide, with the right-eye display, a right-eye alignmentmarker; receive user input to adjust at least one of the left-eye andthe right-eye alignment markers; and vertically adjust image content inthe at least one of the left-eye and right-eye displays based on thereceived user input.

In some embodiments, an augment reality system is provided. Theaugmented reality system comprises a head-mounted display (TIMID)configured to present virtual content by outputting light to a user, animaging system configured to capture images of eyes of the user, and atleast one processor communicatively coupled to the HMD and the imagingdevice. The at least one processor is configured to determine aninterocular axis of the user that extends between the user's left andright eyes based at least in part on one or more images captured by theimaging system; determine an orientation of the HMD relative to theinterocular axis of the user; and provide the user with feedback basedon the determined orientation of the HMD relative to the interocularaxis of the user.

Additional examples of embodiments are enumerated below.

Example 1. An augmented reality system comprising:

-   -   a head-mounted display configured to present virtual content by        outputting light to a user;    -   an imaging device configured to capture images of eyes of the        user; and    -   at least one processor communicatively coupled to the        head-mounted display and the imaging device, the at least one        processor configured to:        -   provide, with the left-eye display, a left-eye alignment            marker;        -   provide, with the right-eye display, a right-eye alignment            marker;        -   receive user input to adjust at least one of the left-eye            and the right-eye alignment markers; and        -   vertically adjust image content in the at least one of the            left-eye and right-eye displays based on the received user            input.

Example 2. The augmented reality system of Example 1, wherein theprocessor is configured to:

-   -   determine if the head-mounted display system is level relative        to the user's left and right eyes; and    -   provide the user with feedback on whether the head-mounted        display system is level relative to the user's left and right        eyes before providing, with the left-eye display, a left-eye        alignment marker.

Example 3. The augmented reality system of Example 1, wherein theprocessor is configured to provide the user with feedback on whether thehead-mounted display is level relative to the user's left and right eyesby presenting a level marker having an orientation that changes inrelation to an orientation of the head-mounted display relative to theuser's left and right eyes.

Example 4. The augmented reality system of Example 1, further comprisingan eye-tracking system, wherein the processor is configured to determineif the head-mounted display system is level relative to the user's leftand right eyes based on eye tracking data from the eye-tracking system.

Example 5. The augmented reality system of Example 1, further comprisingan eye-tracking system, wherein the processor is further configured todetermine an interocular axis of the user that extends between theuser's left and right eyes based on eye tracking data from theeye-tracking system.

Example 6. The augmented reality system of Example 5, wherein theprocessor is further configured to determine if the head-mounted displaysystem is level relative to the user's left and right eyes bydetermining an orientation of the eye-tracking system relative to theinterocular axis of the user.

Example 7. The augmented reality system of Example 5, wherein theprocessor is configured to provide the user with feedback on whether thehead-mounted display is level relative to the user's left and right eyesby presenting a level marker having an orientation that changes inrelation with the orientation of the eye-tracking system relative to theinterocular axis of the user.

Example 8. The augmented reality system of Example 5, wherein theprocessor is configured to provide the user with feedback on whether thehead-mounted display is level relative to the user's left and right eyesby presenting a static level marker associated with an orientation ofthe head-mounted display and presenting a dynamic level markerassociated with an orientation of the interocular axis, wherein thedynamic level marker moves relative to the static level marker as theorientation of the interocular axis changes relative to the head-mounteddisplay.

Example 9. The augmented reality system of Example 8, wherein thedynamic level marker merges with the static level marker when thehead-mounted display is level relative to the interocular axis.

Example 10. The augmented reality system of Example 1, wherein theleft-eye alignment marker comprises a first horizontal line and whereinthe right-eye alignment marker comprises a second horizontal line.

Example 11. The augmented reality system of Example 10, wherein theprocessor is configured to receive user input to adjust at least one ofthe left-eye and right-eye alignment markers in the form of user inputto raise or lower at least one of the first and second horizontal lines.

Example 12. The augmented reality system of Example 10, wherein theprocessor is configured to receive user input to adjust at least one ofthe left-eye and right-eye alignment markers in the form of user inputto raise or lower at least one of the first and second horizontal linesuntil the first and second horizontal lines are level from theperspective of the user.

Example 13. The augmented reality system of Example 1, wherein thehead-mounted display comprises a first waveguide stack configured topass light from the world into the left eye of the user and a secondwaveguide stack configured to pass light from the world into the righteye of the user and wherein each waveguide stack comprises a pluralityof waveguides.

Example 14. The augmented reality system of Example 1, wherein thehead-mounted display comprises a first waveguide stack configured topass light from the world into the left eye of the user and a secondwaveguide stack configured to pass light from the world into the righteye of the user,

-   -   wherein each waveguide stack comprises a plurality of waveguides        where one or more waveguides of that waveguide stack are        configured to output light to the user with a different amount        of wavefront divergence than one or more other waveguides of        that waveguide stack,    -   wherein different amounts of wavefront divergence are associated        with different accommodation by the eye, and    -   wherein the outputted light with different amounts of wavefront        divergence forms virtual objects at different perceived depths        away from the user.

Example 15. A method for vertically aligning, with a user's left andright eyes, displayed content of a left-eye display and a right-eyedisplay of a head-mounted display system, the method comprising:

-   -   providing, with the left-eye display, a left-eye alignment        marker;    -   providing, with the right-eye display, a right-eye alignment        marker;    -   receiving user input to adjust at least one of the left-eye and        the right-eye alignment markers; and    -   vertically adjusting image content in at least one of the        left-eye and right-eye displays based on the received user        input.

Example 16. The method of Example 15, further comprising:

-   -   determining a level of the head-mounted display system relative        to the user's left and right eyes; and    -   providing the user feedback on whether the head-mounted display        system is level relative to the user's left and right eyes        before providing the left or right-eye alignment markers.

Example 17. The method of Example 15, wherein the head-mounted displaysystem comprises an eye-tracking system, wherein determining the levelof the head-mounted display system relative to the user's left and righteyes comprises utilizing the eye-tracking system to determine aninterocular axis of the user that extends between the user's left andright eyes and determining a level of the head-mounted display systemrelative to the interocular axis.

Example 18. The method of Example 17, wherein utilizing the eye-trackingsystem to determine an interocular axis of the user comprises:

-   -   determining, with the eye-tracking system, a center of rotation        of the user's left eye;    -   determining, with the eye-tracking system, a center of rotation        of the user's right eye; and    -   determining a position of a line extending between the centers        of rotation of the user's left and right eyes, wherein the line        constitutes the interocular axis.

Example 19. The method of Example 15, wherein providing the left-eyealignment marker comprises providing, with the left-eye display, a firstvertical alignment marker and a first horizontal alignment marker andwherein providing the right-eye alignment marker comprises providing,with the right-eye display, a second vertical alignment marker and asecond horizontal alignment marker.

Example 20. The method of Example 19, wherein, when viewed by the user,the first and second vertical alignment markers fuse together in theuser's vision and the first and second horizontal alignment markersremain unfused in the user's vision.

Example 21. The method of Example 19, receiving the user input to adjustthe at least one of the left-eye and right-eye alignment markerscomprises receiving user input to move at least one of the first andsecond horizontal alignment markers vertically.

Example 22. The method of Example 19, receiving the user input to adjustthe at least one of the left-eye and right-eye alignment markerscomprises receiving user input to move at least one of the first andsecond horizontal alignment markers vertically until the first andsecond horizontal alignment markers are vertically aligned with eachother in the user's vision.

Example 23. The method of Example 15, further comprising:

-   -   determining that the user has worn the head-mounted display        system for a given threshold of time; and    -   in response to determining that the user has worn the        head-mounted display system for the given threshold of time,        performing determining the level of the head-mounted display        system.

Example 24. An augmented reality system comprising:

-   -   a head-mounted display configured to present virtual content by        outputting light to a user, the head-mounted display comprising        a left-eye display configured to present virtual content to the        user's left eye and a right-eye display configured to present        virtual content to the user's right eye; and    -   at least one processor communicatively coupled to the        head-mounted display, the at least one processor configured to:        -   provide, with the left-eye display, a left-eye alignment            marker;        -   provide, with the right-eye display, a right-eye alignment            marker;        -   receive user input to adjust at least one of the left-eye            and the right-eye alignment markers; and        -   vertically adjust image content in the at least one of the            left-eye and right-eye displays based on the received user            input.

Example 25. The augmented reality system of Example 24, wherein theleft-eye alignment marker comprises a first horizontal line and whereinthe right-eye alignment marker comprises a second horizontal line.

Example 26. The augmented reality system of Example 25, wherein theprocessor is configured to receive user input to adjust at least one ofthe left-eye and right-eye alignment markers in the form of user inputto raise or lower at least one of the first and second horizontal lines.

Example 27. The augmented reality system of Example 25, wherein theprocessor is configured to receive user input to adjust at least one ofthe left-eye and right-eye alignment markers in the form of user inputto raise or lower at least one of the first and second horizontal linesuntil the first and second horizontal lines are level from theperspective of the user.

Example 28. The augmented reality system of Example 24, wherein theleft-eye display comprises a first waveguide stack configured to passlight from the world into the left eye of the user and the right-eyedisplay comprises a second waveguide stack configured to pass light fromthe world into the right eye of the user, and wherein each waveguidestack comprises a plurality of waveguides.

Example 29. The augmented reality system of Example 24, wherein theleft-eye display comprises a first waveguide stack configured to passlight from the world into the left eye of the user and the right-eyedisplay comprises a second waveguide stack configured to pass light fromthe world into the right eye of the user,

-   -   wherein each waveguide stack comprises a plurality of waveguides        where one or more waveguides of that waveguide stack are        configured to output light to the user with a different amount        of wavefront divergence than one or more other waveguides of        that waveguide stack,    -   wherein different amounts of wavefront divergence are associated        with different accommodation by the eye, and    -   wherein the outputted light with different amounts of wavefront        divergence forms virtual objects at different perceived depths        away from the user.

Example 30. The augmented reality system of Example 24, wherein toprovide, with the left-eye display and the right-eye display, theleft-eye alignment marker and the right-eye alignment marker,respectively, the at least one processor is configured to:

-   -   provide, with the left-eye display, a first vertical alignment        marker and a first horizontal alignment marker; and    -   provide, with the right-eye display, a second vertical alignment        marker and a second horizontal alignment marker.

Example 31. The augmented reality system of Example 30, wherein, whenviewed by the user, the first and second vertical alignment markers fusetogether in the user's vision and the first and second horizontalalignment markers remain unfused in the user's vision.

Example 32. The augmented reality system of Example 30, wherein toreceive user input to adjust at least one of the left-eye and theright-eye alignment markers, the at least one processor is configuredto:

-   -   receive user input to move at least one of the first and second        horizontal alignment markers vertically.

Example 33. The augmented reality system of Example 30, wherein toreceive user input to adjust at least one of the left-eye and theright-eye alignment markers, the at least one processor is configuredto:

-   -   receive user input to move at least one of the first and second        horizontal alignment markers vertically until the first and        second horizontal alignment markers are vertically aligned with        each other in the user's vision.

Example 34. The augmented reality system of Example 24, wherein the atleast one processor is further configured to:

-   -   select a first vertical position at which to present the        left-eye alignment marker; and    -   select a second vertical position at which to present the        right-eye alignment marker, and    -   wherein to provide, with the left-eye display and the right-eye        display, the left-eye alignment marker and the right-eye        alignment marker, respectively, the at least one processor is        configured to:        -   provide, with the left-eye display, the left-eye alignment            marker at the first vertical position; and        -   provide, with the right-eye display, the right-eye alignment            marker at the second vertical position.

Example 35. The augmented reality system of Example 24, wherein toselect the first vertical position at which to present the left-eyealignment marker and select the second vertical position at which topresent the right-eye alignment marker, the at least one processor isconfigured to:

-   -   pseudo- or quasi-randomly select the first vertical position at        which to present the left-eye alignment marker; and    -   pseudo- or quasi-randomly select the second vertical position at        which to present the right-eye alignment marker.

Example 36. The augmented reality system of Example 24, wherein tovertically adjust image content in the at least one of the left-eye andright-eye displays based on the received user input, the at least oneprocessor is configured to:

-   -   vertically adjust image content in the at least one of the        left-eye and right-eye displays based on the received user        input, the first vertical position selected, and the second        vertical position selected.

Example 37. The augmented reality system of Example 24, wherein tovertically adjust image content in the at least one of the left-eye andright-eye displays based on the received user input, the at least oneprocessor is configured to:

-   -   adjust one or more extrinsic parameters of at least one of a        first render camera that is associated with the left-eye display        and a second render camera that is associated with the right-eye        display.

Example 38. The augmented reality system of Example 37, wherein the oneor more extrinsic parameters include at least one of a position and anorientation.

Example 39. The augmented reality system of Example 24, furthercomprising an imaging system configured to capture images of eyes of theuser, and

-   -   wherein the at least one processor is communicatively coupled to        the imaging system, the at least one processor further        configured to:        -   determine whether the head-mounted display is level relative            to the user's left and right eyes based at least in part on            one or more images captured by the imaging system.

Example 40. The augmented reality system of Example 39, wherein the atleast one processor is configured to provide, with the left-eye displayand the right-eye display, the left-eye alignment marker and theright-eye alignment marker, respectively, in response to a determinationthat the head-mounted display is level relative to the user's left andright eyes.

Example 41. The augmented reality system of Example 24, wherein theleft-eye display comprises a first waveguide stack and the right-eyedisplay comprises a second waveguide stack, each of which comprises aplurality of waveguides configured to output light to the user, and

-   -   wherein to provide, with the left-eye display and the right-eye        display, the left-eye alignment marker and the right-eye        alignment marker, respectively, the at least one processor is        configured to:        -   provide, with a single one of the plurality of waveguides in            the first waveguide stack, the left-eye alignment marker;            and        -   provide, with a single one of the plurality of waveguides in            the second waveguide stack, the right-eye alignment marker.

Example 42. An augmented reality system comprising:

-   -   a head-mounted display configured to present virtual content by        outputting light to a user;    -   an imaging system configured to capture images of eyes of the        user; and    -   at least one processor communicatively coupled to the        head-mounted display and the imaging system, the at least one        processor configured to:        -   determine an interocular axis of the user that extends            between the user's left and right eyes based at least in            part on one or more images captured by the imaging system;        -   determine an orientation of the head-mounted display            relative to the interocular axis of the user; and        -   provide the user with feedback based on the determined            orientation of the head-mounted display relative to the            interocular axis of the user.

Example 43. The augmented reality system of Example 42, wherein toprovide the user with feedback based on the determined orientation ofthe head-mounted display relative to the interocular axis of the user,the at least one processor is configured to:

-   -   present a level marker having an orientation that changes in        relation to the determined orientation of the head-mounted        display relative to the user's left and right eyes.

Example 44. The augmented reality system of Example 42, wherein toprovide the user with feedback based on the determined orientation ofthe head-mounted display relative to the interocular axis of the user,the at least one processor is configured to:

-   -   present a static level marker associated with an orientation of        the head-mounted display; and    -   present a dynamic level marker associated with an orientation of        the interocular axis, wherein the dynamic level marker moves        relative to the static level marker as the orientation of the        interocular axis changes relative to the head-mounted display.

Example 45. The augmented reality system of Example 44, wherein thedynamic level marker merges with the static level marker when thehead-mounted display is level relative to the interocular axis.

Example 46. The augmented reality system of Example 42, wherein todetermine the interocular axis of the user that extends between theuser's left and right eyes based at least in part on one or more imagescaptured by the imaging system, the at least one processor is configuredto:

-   -   determine, based at least in part on one or more images of the        user's left eye captured by the imaging system, a center of        rotation of the user's left eye;    -   determine, based at least in part on one or more images of the        user's right eye captured by the imaging system, a center of        rotation of the user's right eye; and    -   determine a position of a line extending between the centers of        rotation of the user's left and right eyes, wherein the line        constitutes the interocular axis of the user.

Example 47. The augmented reality system of Example 42, wherein the atleast one processor is further configured to:

-   -   determine whether the user has worn the head-mounted display for        at least a threshold of amount of time, and    -   wherein the at least one processor is configured to determine        the interocular axis of the user that extends between the user's        left and right eyes based at least in part on one or more images        captured by the imaging system in response to a determination        that the user has worn the head-mounted display for at least the        threshold of amount of time.

Example 48. The augmented reality system of Example 42, wherein the atleast one processor is further configured to:

-   -   determine whether the head-mounted display is level relative to        the user's left and right eyes based on the determined        orientation of the head-mounted display relative to the        interocular axis of the user.

Example 47. The augmented reality system of Example 46, wherein the atleast one processor is further configured to, in response to adetermination that the head-mounted display is level relative to theuser's left and right eyes:

-   -   provide, with a left-eye display of the head-mounted display, a        left-eye alignment marker;    -   provide, with a right-eye display of the head-mounted display, a        right-eye alignment marker;    -   receive user input to adjust at least one of the left-eye and        the right-eye alignment markers; and    -   vertically adjust image content in the at least one of the        left-eye and right-eye displays based on the received user        input.

Example 48. The augmented reality system of Example 46, wherein toprovide the user with feedback based on the determined orientation ofthe head-mounted display relative to the interocular axis of the user,the at least one processor is configured to:

-   -   provide the user with feedback indicating whether the        head-mounted display is level relative to the user's left and        right eye.

Details of one or more embodiments of the subject matter described inthis specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description limit the scope ofthe inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates example components of a wearablesystem.

FIG. 4 schematically illustrates an example of a waveguide stack of awearable device for outputting image information to a user.

FIG. 5 schematically illustrates an example of an eye and schematicallyillustrates an example coordinate system for determining an eye pose ofan eye.

FIG. 6 is a schematic diagram of a wearable system that includes an eyetracking system.

FIG. 7A is a block diagram of a wearable system that may include an eyetracking system.

FIG. 7B is a block diagram of a render controller in a wearable system.

FIG. 7C is a block diagram of a registration observer in a head-mounteddisplay system.

FIG. 8A is a schematic diagram of an eye showing the eye's cornealsphere.

FIG. 8B illustrates an example corneal glint detected by an eye-trackingcamera.

FIGS. 8C-8E illustrate example stages of locating a user's cornealcenter with an eye tracking module in a wearable system.

FIGS. 9A-9C illustrate an example normalization of the coordinate systemof eye tracking images.

FIGS. 9D-9G illustrate example stages of locating a user's pupil centerwith an eye tracking module in a wearable system.

FIG. 10 illustrates an example of an eye including the eye's optical andvisual axes and the eye's center of rotation.

FIG. 11 is a process flow diagram of an example of a method for usingeye tracking in rendering content and providing feedback on registrationin a wearable device.

FIGS. 12A and 12B illustrate a nominal position of a display elementrelative to a user's eye and illustrate a coordinate system fordescribing the positions of the display element and the user's eyerelative to one another.

FIG. 13 illustrates an example display screen that may be provided to auser of a head-mounted display system as part of leveling thehead-mounted display system on the user's head.

FIG. 14 illustrates an example display screen that may be provided tothe user of the head-mounted display system of FIG. 13 after the userhas leveled the head-mounted display system on their head.

FIG. 15 illustrates an example display screen that may be provided to auser of a head-mounted display system as part of adjusting verticalalignment between left and right eye displays of the display system.

FIG. 16 illustrates examples of left-eye, right-eye, and fused imagesthat may be provided to the user of FIG. 15 as part of adjusting thevertical alignment between the left and right displays of the displaysystem.

FIG. 17 is a perspective view of a user's hand and a controllerincluding various input devices.

FIG. 18 is a flow diagram illustrative of an embodiment of leveling ahead-mounted display system on a user's head and of adjusting a verticalalignment of left and right displays of the display system.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION

Display systems may include a head-mounted display (HMD) which maydisplay virtual objects such that the objects appear to be locatedwithin the user's ambient environment. In some embodiments, the virtualobjects may be perceived by the user to be three-dimensional (3D).

The HMD may include two or more distinct displays or a single contiguousdisplay. In any case, different images may be outputted to the left andright eyes of the user. As used herein, references to left-eye andright-eye displays or displays associated with the right and the lefteyes of the viewer may be understood to indicate a display deviceconfigured to output different images to the left and right eyes of theviewer. Thus, left-eye and right-eye displays may indicate displays thatare physically separated, or a single display configured to outputdifferent images to each eye. In some embodiments, the different imagesmay be stereoscopic images that present slightly different views of thesame object or scene, with the different views being fused together bythe human visual system, which can create the perception of depth.

In some situations, the left and right displays may not be properlyvertically aligned relative to one another and a user's eyes. Variousfactors may combine to determine the positions of the left and right eyedisplays relative to one another and to the user's eyes. For example,the HMD may become deformed or warped over time, leading to left-eyedisplay, right-eye display vertical misalignments. The HMD can bedeformed or become warped as the result of numerous factors includingthermal cycling, physical shock (e.g., due to being dropped), elastic orinelastic bending from rough handling, aging of materials, etc. In somecases, one of the left or right-eye displays may become rotated and/orvertically translated relative to the other of the left or right-eyedisplays.

Such vertical misalignment may degrade the user experience and possiblycause user discomfort. Without being limited by theory, it is believedthat the human visual system is not accustomed to receiving verticallymisaligned light for forming corresponding left and right eyestereoscopic images, where the light propagating towards one eye may bevertically offset from light propagating towards the other eye to formthe corresponding stereoscopic images. It will be appreciated that lightpropagating towards each eye may form stereoscopic images specific tothose eyes. As an example, if light from one display is directedhorizontally to reach one eye and light from another display is directeddownwards to reach the other eye, then these displays may be consideredvertically misaligned. It is believed that even relatively smallvertical misalignments between the left and right eyes may causediscomfort such as harmful eye strain and headaches.

To reduce viewer discomfort due to vertical misalignment, differences invertical alignment between a left-eye display and a right-eye displaymay be reduced. For example, the left-eye display and the right-eyedisplay may be aligned vertically by physically moving those displays.In some cases, such physical movement of the displays is not practicalor desirable. Advantageously, in some embodiments, vertical alignmentbetween a left-eye display and a right-eye display is effectivelyachieved by causing the displays to shift image content vertically up ordown on one or both of the displays. In some embodiments, the imageshifting may allow the left-eye and right-eye displays to have similarvertical relationships with their corresponding left and right eyes.

Various systems and techniques described herein are at least in partdirected to solving problems related to proper vertical alignment ofleft-eye and right-eye displays, which may allow the viewer to viewimage content comfortably. In some embodiments, a head-mounted displaysystem may be configured to determine the positions of the eyes of theuser (e.g., using a component such as an inward-facing imaging system,which may be an eye tracking system). Determining the position of theeyes may include determining the positions of representative pointsassociated with the eyes such as the respective centers of rotation ofthe eyes and the position and/or orientation of the user's interocularaxis (e.g., an axis extending between corresponding parts of the user'sleft and right eyes, such as an axis between the center of rotation ofthe user's left eye and the center of rotation of the user's right eye).The display system may then determine whether the HMD is level withrespect to the user's eyes (e.g., level with the user's interocularaxis). After leveling, the display system may present the user's lefteye with a left-eye alignment marker and the user's right eye with aright-eye alignment marker. Preferably, the alignment markers includehorizontal components that are not fused by the human visual system andwhich make apparent in vertical misalignments between the left-eye andright-eye displays. The display system may then solicit feedback fromthe user that identifies and/or adjusts vertical alignment differencesbetween the left and right-eye displays (e.g., by asking the user toalign the left-eye and right-eye alignment markers). Using suchfeedback, the display system may adjust content displayed through thesystem by compensating for any vertical alignment differences identifiedby the user, thus improving the user's comfort when viewing the HMD.

In some embodiments, the display system includes a plurality ofwaveguides formed in a stack for outputting image information to a user.An alignment process, as discussed herein, may be conducted for eachwaveguide. For example, different waveguides may be configured to outputdifferent component colors and/or different amounts of wavefrontdivergence corresponding to different depth planes, and alignments maybe performed with each component color (for each of the correspondingcomponent color waveguides) and/or each depth plane (for each waveguidewith a corresponding amount of wavefront divergence).

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout. Unless indicated otherwise, thedrawings are schematic and not necessarily drawn to scale.

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as a head-mounted displaysystem or as an augmented reality (AR) system) may be configured topresent 2D or 3D virtual images to a user. The images may be stillimages, frames of a video, or a video, in combination or the like. Atleast a portion of the wearable system may be implemented on a wearabledevice that may present a VR, AR, or MR environment, alone or incombination, for user interaction. The wearable device may be usedinterchangeably as an AR device (ARD). Further, for the purpose of thepresent disclosure, the term “AR” is used interchangeably with the term“MR”.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1 , an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences may be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues provide credible perceptionsof depth.

FIG. 2 illustrates an example of wearable system 200 which may beconfigured to provide an AR/VR/MR scene. The wearable system 200 mayalso be referred to as the AR system 200. The wearable system 200includes a display 220, and various mechanical and electronic modulesand systems to support the functioning of display 220. The display 220may be coupled to a frame 230, which is wearable by a user, wearer, orviewer 210. The display 220 may be positioned in front of the eyes ofthe user 210. The display 220 may present AR/VR/MR content to a user.Because the display 220 may be worn on the head of the user 210, it mayalso be referred to as a head-mounted display (HMD) and the wearablesystem 200, comprising the display 220, may also be referred to as ahead-mounted display system.

In some embodiments, a speaker 240 is coupled to the frame 230 andpositioned adjacent the ear canal of the user (in some embodiments,another speaker, not shown, is positioned adjacent the other ear canalof the user to provide for stereo/shapeable sound control). The display220 may include an audio sensor (e.g., a microphone) 232 for detectingan audio stream from the environment and capture ambient sound. In someembodiments, one or more other audio sensors, not shown, are positionedto provide stereo sound reception. Stereo sound reception may be used todetermine the location of a sound source. The wearable system 200 mayperform voice or speech recognition on the audio stream.

The wearable system 200 may include an outward-facing imaging system 464(shown in FIG. 4 ) which observes the world in the environment aroundthe user. The wearable system 200 may also include an inward-facingimaging system 462 (shown in FIG. 4 ) which may track the eye movementsof the user. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210. The inward-facing imaging system 462 mayinclude one or more cameras. For example, at least one camera may beused to image each eye. The images acquired by the cameras may be usedto determine pupil size or eye pose for each eye separately, therebyallowing presentation of image information to each eye to be dynamicallytailored to that eye.

As an example, the wearable system 200 may use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video.

The display 220 may be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), audio sensors (e.g., microphones),inertial measurement units (IMUs), accelerometers, compasses, globalpositioning system (GPS) units, radio devices, or gyroscopes; or b)acquired or processed using remote processing module 270 or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 or 264, such as via wiredor wireless communication links, to the remote processing module 270 orremote data repository 280 such that these remote modules are availableas resources to the local processing and data module 260. In addition,remote processing module 280 and remote data repository 280 may beoperatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

Example Components of a Wearable System

FIG. 3 schematically illustrates example components of a wearablesystem. FIG. 3 shows a wearable system 200 which may include a display220 and a frame 230. A blown-up view 202 schematically illustratesvarious components of the wearable system 200. In certain implements,one or more of the components illustrated in FIG. 3 may be part of thedisplay 220. The various components alone or in combination may collecta variety of data (such as e.g., audio or visual data) associated withthe user of the wearable system 200 or the user's environment. It shouldbe appreciated that other embodiments may have additional or fewercomponents depending on the application for which the wearable system isused. Nevertheless, FIG. 3 provides a basic idea of some of the variouscomponents and types of data that may be collected, analyzed, and storedthrough the wearable system.

FIG. 3 shows an example wearable system 200 which may include thedisplay 220. The display 220 may comprise a display lens 226 that may bemounted to a user's head or a housing or frame 230, which corresponds tothe frame 230. The display lens 226 may comprise one or more transparentmirrors positioned by the housing 230 in front of the user's eyes 302,304 and may be configured to bounce projected light 338 into the eyes302, 304 and facilitate beam shaping, while also allowing fortransmission of at least some light from the local environment. Thewavefront of the projected light beam 338 may be bent or focused tocoincide with a desired focal distance of the projected light. Asillustrated, two wide-field-of-view machine vision cameras 316 (alsoreferred to as world cameras) may be coupled to the housing 230 to imagethe environment around the user. These cameras 316 may be dual capturevisible light/non-visible (e.g., infrared) light cameras. The cameras316 may be part of the outward-facing imaging system 464 shown in FIG. 4. Image acquired by the world cameras 316 may be processed by the poseprocessor 336. For example, the pose processor 336 may implement one ormore object recognizers 708 (e.g., shown in FIG. 7 ) to identify a poseof a user or another person in the user's environment or to identify aphysical object in the user's environment.

With continued reference to FIG. 3 , a pair of scanned-lasershaped-wavefront (e.g., for depth) light projector modules with displaymirrors and optics configured to project light 338 into the eyes 302,304 are shown. The depicted view also shows two miniature infraredcameras 324 paired with infrared light sources 326 (such as lightemitting diodes “LED”s), which are configured to be able to track theeyes 302, 304 of the user to support rendering and user input. Thecameras 324 may be part of the inward-facing imaging system 462 shown inFIG. 4 . The wearable system 200 may further feature a sensor assembly339, which may comprise X, Y, and Z axis accelerometer capability aswell as a magnetic compass and X, Y, and Z axis gyro capability,preferably providing data at a relatively high frequency, such as 200Hz. The sensor assembly 339 may be part of the IMU described withreference to FIG. 2A The depicted system 200 may also comprise a headpose processor 336, such as an ASIC (application specific integratedcircuit), FPGA (field programmable gate array), or ARM processor(advanced reduced-instruction-set machine), which may be configured tocalculate real or near-real time user head pose from wide field of viewimage information output from the capture devices 316. The head poseprocessor 336 may be a hardware processor and may be implemented as partof the local processing and data module 260 shown in FIG. 2A.

The wearable system may also include one or more depth sensors 234. Thedepth sensor 234 may be configured to measure the distance between anobject in an environment to a wearable device. The depth sensor 234 mayinclude a laser scanner (e.g., a lidar), an ultrasonic depth sensor, ora depth sensing camera. In certain embodiments, where the cameras 316have depth sensing ability, the cameras 316 may also be considered asdepth sensors 234.

Also shown is a processor 332 configured to execute digital or analogprocessing to derive pose from the gyro, compass, or accelerometer datafrom the sensor assembly 339. The processor 332 may be part of the localprocessing and data module 260 shown in FIG. 2 . The wearable system 200as shown in FIG. 3 may also include a position system such as, e.g., aGPS 337 (global positioning system) to assist with pose and positioninganalyses. In addition, the GPS may further provide remotely-based (e.g.,cloud-based) information about the user's environment. This informationmay be used for recognizing objects or information in user'senvironment.

The wearable system may combine data acquired by the GPS 337 and aremote computing system (such as, e.g., the remote processing module270, another user's ARD, etc.) which may provide more information aboutthe user's environment. As one example, the wearable system maydetermine the user's location based on GPS data and retrieve a world map(e.g., by communicating with a remote processing module 270) includingvirtual objects associated with the user's location. As another example,the wearable system 200 may monitor the environment using the worldcameras 316 (which may be part of the outward-facing imaging system 464shown in FIG. 4 ). Based on the images acquired by the world cameras316, the wearable system 200 may detect objects in the environment(e.g., by using one or more object recognizers 708 shown in FIG. 7 ).The wearable system may further use data acquired by the GPS 337 tointerpret the characters.

The wearable system 200 may also comprise a rendering engine 334 whichmay be configured to provide rendering information that is local to theuser to facilitate operation of the scanners and imaging into the eyesof the user, for the user's view of the world. The rendering engine 334may be implemented by a hardware processor (such as, e.g., a centralprocessing unit or a graphics processing unit). In some embodiments, therendering engine is part of the local processing and data module 260.The rendering engine 334 may be communicatively coupled (e.g., via wiredor wireless links) to other components of the wearable system 200. Forexample, the rendering engine 334, may be coupled to the eye cameras 324via communication link 274, and be coupled to a projecting subsystem 318(which may project light into user's eyes 302, 304 via a scanned laserarrangement in a manner similar to a retinal scanning display) via thecommunication link 272. The rendering engine 334 may also be incommunication with other processing units such as, e.g., the sensor poseprocessor 332 and the image pose processor 336 via links 276 and 294respectively.

The cameras 324 (e.g., mini infrared cameras) may be utilized to trackthe eye pose to support rendering and user input. Some example eye posesmay include where the user is looking or at what depth he or she isfocusing (which may be estimated with eye vergence). The GPS 337, gyros,compass, and accelerometers 339 may be utilized to provide coarse orfast pose estimates. One or more of the cameras 316 may acquire imagesand pose, which in conjunction with data from an associated cloudcomputing resource, may be utilized to map the local environment andshare user views with others.

The example components depicted in FIG. 3 are for illustration purposesonly. Multiple sensors and other functional modules are shown togetherfor ease of illustration and description. Some embodiments may includeonly one or a subset of these sensors or modules. Further, the locationsof these components are not limited to the positions depicted in FIG. 3. Some components may be mounted to or housed within other components,such as a belt-mounted component, a hand-held component, or a helmetcomponent. As one example, the image pose processor 336, sensor poseprocessor 332, and rendering engine 334 may be positioned in a beltpackand configured to communicate with other components of the wearablesystem via wireless communication, such as ultra-wideband, Wi-Fi,Bluetooth, etc., or via wired communication. The depicted housing 230preferably is head-mountable and wearable by the user. However, somecomponents of the wearable system 200 may be worn to other portions ofthe user's body. For example, the speaker 240 may be inserted into theears of a user to provide sound to the user.

Regarding the projection of light 338 into the eyes 302, 304 of theuser, in some embodiment, the cameras 324 may be utilized to measurewhere the centers of a user's eyes are geometrically verged to, which,in general, coincides with a position of focus, or “depth of focus”, ofthe eyes. A 3-dimensional surface of all points the eyes verge to may bereferred to as the “horopter”. The focal distance may take on a finitenumber of depths, or may be infinitely varying. Light projected from thevergence distance appears to be focused to the subject eye 302, 304,while light in front of or behind the vergence distance is blurred.Examples of wearable devices and other display systems of the presentdisclosure are also described in U.S. Patent Publication No.2016/0270656, which is incorporated by reference herein in its entirety.

The human visual system is complicated and providing a realisticperception of depth is challenging. Viewers of an object may perceivethe object as being three-dimensional due to a combination of vergenceand accommodation. Vergence movements (e.g., rolling movements of thepupils toward or away from each other to converge the lines of sight ofthe eyes to fixate upon an object) of the two eyes relative to eachother are closely associated with focusing (or “accommodation”) of thelenses of the eyes. Under normal conditions, changing the focus of thelenses of the eyes, or accommodating the eyes, to change focus from oneobject to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex.” Likewise, achange in vergence will trigger a matching change in accommodation,under normal conditions. Display systems that provide a better matchbetween accommodation and vergence may form more realistic andcomfortable simulations of three-dimensional imagery.

Further spatially coherent light with a beam diameter of less than about0.7 millimeters may be correctly resolved by the human eye regardless ofwhere the eye focuses. Thus, to create an illusion of proper focaldepth, the eye vergence may be tracked with the cameras 324, and therendering engine 334 and projection subsystem 318 may be utilized torender all objects on or close to the horopter in focus, and all otherobjects at varying degrees of defocus (e.g., using intentionally-createdblurring). Preferably, the system 220 renders to the user at a framerate of about 60 frames per second or greater. As described above,preferably, the cameras 324 may be utilized for eye tracking, andsoftware may be configured to pick up not only vergence geometry butalso focus location cues to serve as user inputs. Preferably, such adisplay system is configured with brightness and contrast suitable forday or night use.

In some embodiments, the display system preferably has latency of lessthan about 20 milliseconds for visual object alignment, less than about0.1 degree of angular alignment, and about 1 arc minute of resolution,which, without being limited by theory, is believed to be approximatelythe limit of the human eye. The display system 220 may be integratedwith a localization system, which may involve GPS elements, opticaltracking, compass, accelerometers, or other data sources, to assist withposition and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (e.g., such information would facilitate the glasses toknow where they are with respect to the real world).

In some embodiments, the wearable system 200 is configured to displayone or more virtual images based on the accommodation of the user'seyes. Unlike prior 3D display approaches that force the user to focuswhere the images are being projected, in some embodiments, the wearablesystem is configured to automatically vary the focus of projectedvirtual content to allow for a more comfortable viewing of one or moreimages presented to the user. For example, if the user's eyes have acurrent focus of 1 m, the image may be projected to coincide with theuser's focus. If the user shifts focus to 3 m, the image is projected tocoincide with the new focus. Thus, rather than forcing the user to apredetermined focus, the wearable system 200 of some embodiments allowsthe user's eye to a function in a more natural manner.

Such a wearable system 200 may eliminate or reduce the incidences of eyestrain, headaches, and other physiological symptoms typically observedwith respect to virtual reality devices. To achieve this, variousembodiments of the wearable system 200 are configured to project virtualimages at varying focal distances, through one or more variable focuselements (VFEs). In one or more embodiments, 3D perception may beachieved through a multi-plane focus system that projects images atfixed focal planes away from the user. Other embodiments employ variableplane focus, wherein the focal plane is moved back and forth in thez-direction to coincide with the user's present state of focus.

In both the multi-plane focus systems and variable plane focus systems,wearable system 200 may employ eye tracking to determine a vergence ofthe user's eyes, determine the user's current focus, and project thevirtual image at the determined focus. In other embodiments, wearablesystem 200 comprises a light modulator that variably projects, through afiber scanner, or other light generating source, light beams of varyingfocus in a raster pattern across the retina. Thus, the ability of thedisplay of the wearable system 200 to project images at varying focaldistances not only eases accommodation for the user to view objects in3D, but may also be used to compensate for user ocular anomalies, asfurther described in U.S. Patent Publication No. 2016/0270656, which isincorporated by reference herein in its entirety. In some otherembodiments, a spatial light modulator may project the images to theuser through various optical components. For example, as describedfurther below, the spatial light modulator may project the images ontoone or more waveguides, which then transmit the images to the user.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 4400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2, with FIG. 4 schematically showing some parts of that wearable system200 in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2 .

With continued reference to FIG. 4 , the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 or 270 (illustrated in FIG. 2 ) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top or bottom major surfaces, or maybe disposed directly in the volume of the waveguides 440 b, 438 b, 436b, 434 b, 432 b. In some embodiments, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some other embodiments,the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be a monolithicpiece of material and the light extracting optical elements 440 a, 438a, 436 a, 434 a, 432 a may be formed on a surface or in the interior ofthat piece of material.

With continued reference to FIG. 4 , as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it may reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. (Compensating lens layer 430 and thestacked waveguide assembly 480 as a whole may be configured such thatlight coming from the world 470 is conveyed to the eye 410 atsubstantially the same level of divergence (or collimation) as the lighthad when it was initially received by the stacked waveguide assembly480.) Such a configuration provides as many perceived focal planes asthere are available waveguide/lens pairings. Both the light extractingoptical elements of the waveguides and the focusing aspects of thelenses may be static (e.g., not dynamic or electro-active). In somealternative embodiments, either or both may be dynamic usingelectro-active features.

With continued reference to FIG. 4 , the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information may thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets may be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet may beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 (which may be anembodiment of the local processing and data module 260) may beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 may include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The FOV of the world camera may or may not be the same asthe FOV of a viewer 210 which encompasses a portion of the world 470 theviewer 210 perceives at a given time. For example, in some situations,the FOV of the world camera may be larger than the viewer 210 of theviewer 210 of the wearable system 400. The entire region available forviewing or imaging by a viewer may be referred to as the field of regard(FOR). The FOR may include 4π steradians of solid angle surrounding thewearable system 400 because the wearer may move his body, head, or eyesto perceive substantially any direction in space. In other contexts, thewearer's movements may be more constricted, and accordingly the wearer'sFOR may subtend a smaller solid angle. Images obtained from theoutward-facing imaging system 464 may be used to track gestures made bythe user (e.g., hand or finger gestures), detect objects in the world470 in front of the user, and so forth.

The wearable system 400 may include an audio sensor 232, e.g., amicrophone, to capture ambient sound. As described above, in someembodiments, one or more other audio sensors may be positioned toprovide stereo sound reception useful to the determination of locationof a speech source. The audio sensor 232 may comprise a directionalmicrophone, as another example, which may also provide such usefuldirectional information as to where the audio source is located. Thewearable system 400 may use information from both the outward-facingimaging system 464 and the audio sensor 230 in locating a source ofspeech, or to determine an active speaker at a particular moment intime, etc. For example, the wearable system 400 may use the voicerecognition alone or in combination with a reflected image of thespeaker (e.g., as seen in a mirror) to determine the identity of thespeaker. As another example, the wearable system 400 may determine aposition of the speaker in an environment based on sound acquired fromdirectional microphones. The wearable system 400 may parse the soundcoming from the speaker's position with speech recognition algorithms todetermine the content of the speech and use voice recognition techniquesto determine the identity (e.g., name or other demographic information)of the speaker.

The wearable system 400 may also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 may be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) is determined andassumed to be similar for both eyes of the user. The images obtained bythe inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which may be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs,accelerometers, gyroscopes, etc.

The wearable system 400 may include a user input device 466 by which theuser may input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 may includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller may sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 may be inwired or wireless communication with the wearable system 400.

Other Components of the Wearable System

In many embodiments, the wearable system may include other components inaddition or in alternative to the components of the wearable systemdescribed above. The wearable system may, for example, include one ormore haptic devices or components. The haptic devices or components maybe operable to provide a tactile sensation to a user. For example, thehaptic devices or components may provide a tactile sensation of pressureor texture when touching virtual content (e.g., virtual objects, virtualtools, other virtual constructs). The tactile sensation may replicate afeel of a physical object which a virtual object represents, or mayreplicate a feel of an imagined object or character (e.g., a dragon)which the virtual content represents. In some embodiments, hapticdevices or components may be worn by the user (e.g., a user wearableglove). In some embodiments, haptic devices or components may be held bythe user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain embodiments, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4 ) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, and display systems of the present disclosure are described in U.S.Patent Publication No. 2015/0016777, which is incorporated by referenceherein in its entirety.

Example of an Eye Image

FIG. 5 illustrates an image of an eye 500 with eyelids 504, sclera 508(the “white” of the eye), iris 512, and pupil 516. Curve 516 a shows thepupillary boundary between the pupil 516 and the iris 512, and curve 512a shows the limbic boundary between the iris 512 and the sclera 508. Theeyelids 504 include an upper eyelid 504 a and a lower eyelid 504 b. Theeye 500 is illustrated in a natural resting pose (e.g., in which theuser's face and gaze are both oriented as they would be toward a distantobject directly ahead of the user). The natural resting pose of the eye500 may be indicated by a natural resting direction 520, which is adirection orthogonal to the surface of the eye 500 when in the naturalresting pose (e.g., directly out of the plane for the eye 500 shown inFIG. 5 ) and in this example, centered within the pupil 516.

As the eye 500 moves to look toward different objects, the eye pose willchange relative to the natural resting direction 520. The current eyepose may be determined with reference to an eye pose direction 524,which is a direction orthogonal to the surface of the eye (and centeredin within the pupil 516) but oriented toward the object at which the eyeis currently directed. With reference to an example coordinate systemshown in FIG. 5 , the pose of the eye 500 may be expressed as twoangular parameters indicating an azimuthal deflection and a zenithaldeflection of the eye pose direction 524 of the eye, both relative tothe natural resting direction 520 of the eye. For purposes ofillustration, these angular parameters may be represented as θ(azimuthal deflection, determined from a fiducial azimuth) and#(zenithal deflection, sometimes also referred to as a polardeflection). In some embodiments, angular roll of the eye around the eyepose direction 524 may be included in the determination of eye pose, andangular roll may be included in the following analysis. In some otherembodiments, other techniques for determining the eye pose may be used,for example, a pitch, yaw, and optionally roll system.

An eye image may be obtained from a video using any appropriate process,for example, using a video processing algorithm that may extract animage from one or more sequential frames. The pose of the eye may bedetermined from the eye image using a variety of eye-trackingtechniques. For example, an eye pose may be determined by consideringthe lensing effects of the cornea on light sources that are provided.Any suitable eye tracking technique may be used for determining eye posein the eyelid shape estimation techniques described herein.

Example of an Eye Tracking System

FIG. 6 illustrates a schematic diagram of a wearable, or a head-mounted,display system 600 that includes an eye tracking system. Thehead-mounted display system 600 may, in at least some embodiments,include components located in a head-mounted unit 602 and componentslocated in a non-head-mounted unit 604. Non-head mounted unit 604 maybe, as examples, a belt-mounted component, a hand-held component, acomponent in a backpack, a remote component, etc. Incorporating some ofthe components of the head-mounted display system 600 innon-head-mounted unit 604 may help to reduce the size, weight,complexity, and cost of the head-mounted unit 602. In some embodiments,some or all of the functionality described as being performed by one ormore components of head-mounted unit 602 and/or non-head mounted 604 maybe provided by way of one or more components included elsewhere in thehead-mounted display system 600. For example, some or all of thefunctionality described below in association with a CPU 612 ofhead-mounted unit 602 may be provided by way of a CPU 616 of non-headmounted unit 604, and vice versa. In some examples, some or all of suchfunctionality may be provided by way of peripheral devices ofhead-mounted display system 600. Furthermore, in some embodiments, someor all of such functionality may be provided by way of one or more cloudcomputing devices or other remotely-located computing devices in amanner similar to that which has been described above with reference toFIG. 2 .

As shown in FIG. 6 , head-mounted display system 600 may include an eyetracking system including a camera 324 that captures images of a user'seye 610. If desired, the eye tracking system may also include lightsources 326 a and 326 b (such as light emitting diodes “LED”s). Thelight sources 326 a and 326 b may generate glints (i.e., reflections offof the user's eyes that appear in images of the eye captured by camera324). The positions of the light sources 326 a and 326 b relative to thecamera 324 may be known and, as a consequence, the positions of theglints within images captured by camera 324 may be used in tracking theuser's eyes (as will be discussed in more detail below in connectionwith FIGS. 7-11 ). In at least one embodiment, there may be one lightsource 326 and one camera 324 associated with a single one of the user'seyes 610. In another embodiment, there may be one light source 326 andone camera 324 associated with each of a user's eyes. 610. In yet otherembodiments, there may be one or more cameras 324 and one or more lightsources 326 associated with one or each of a user's eyes 610. As aspecific example, there may be two light sources 326 a and 326 b and oneor more cameras 324 associated with each of a user's eyes 610. Asanother example, there may be three or more light sources such as lightsources 326 a and 326 b and one or more cameras 324 associated with eachof a user's eyes 610.

Eye tracking module 614 may receive images from eye-tracking camera(s)324 and may analyze the images to extract various pieces of information.As examples, the eye tracking module 614 may detect the user's eyeposes, a three-dimensional position of the user's eye relative to theeye-tracking camera 324 (and to the head-mounted unit 602), thedirection one or both of the user's eyes 610 are focused on, the user'svergence depth (i.e., the depth from the user at which the user isfocusing on), the positions of the user's pupils, the positions of theuser's cornea and cornea sphere, the center of rotation of each of theuser's eyes, and the center of perspective of each of the user's eyes.The eye tracking module 614 may extract such information usingtechniques described below in connection with FIGS. 7-11 . As shown inFIG. 6 , eye tracking module 614 may be a software module implementedusing a CPU 612 in a head-mounted unit 602.

Data from eye tracking module 614 may be provided to other components inthe wearable system. As example, such data may be transmitted tocomponents in a non-head-mounted unit 604 such as CPU 616 includingsoftware modules for a light-field render controller 618 and aregistration observer 620, which may be configured to evaluate whetherthe display of the head-mounted display system 600 is properlyregistered with the eyes of the user.

Render controller 618 may use information from eye tracking module 614to adjust images displayed to the user by render engine 622 (e.g., arender engine that may be a software module in GPU 621 and that mayprovide images to display 220). As an example, the render controller 618may adjust images displayed to the user based on the user's center ofrotation or center of perspective. In particular, the render controller618 may use information on the user's center of perspective to simulatea render camera (i.e., to simulate collecting images from the user'sperspective) and may adjust images displayed to the user based on thesimulated render camera.

A “render camera,” which is sometimes also referred to as a “pinholeperspective camera” (or simply “perspective camera”) or “virtual pinholecamera” (or simply “virtual camera”), is a simulated camera for use inrendering virtual image content possibly from a database of objects in avirtual world. The objects may have locations and orientations relativeto the user or wearer and possibly relative to real objects in theenvironment surrounding the user or wearer. In other words, the rendercamera may represent a perspective within render space from which theuser or wearer is to view 3D virtual contents of the render space (e.g.,virtual objects). The render camera may be managed by a render engine torender virtual images based on the database of virtual objects to bepresented to the eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a pinhole camera (corresponding to the“render camera”) having a specific set of intrinsic parameters (e.g.,focal length, camera pixel size, principal point coordinates,skew/distortion parameters, etc.), and a specific set of extrinsicparameters (e.g., translational components and rotational componentsrelative to the virtual world). The virtual images are taken from theperspective of such a camera having a position and orientation of therender camera (e.g., extrinsic parameters of the render camera). Itfollows that the system may define and/or adjust intrinsic and extrinsicrender camera parameters. For example, the system may define aparticular set of extrinsic render camera parameters such that virtualimages may be rendered as if captured from the perspective of a camerahaving a specific location with respect to the user's or wearer's eye soas to provide images that appear to be from the perspective of the useror wearer. The system may later dynamically adjust extrinsic rendercamera parameters on-the-fly so as to maintain registration with thespecific location. Similarly, intrinsic render camera parameters may bedefined and dynamically adjusted over time. In some embodiments, theimages are rendered as if captured from the perspective of a camerahaving an aperture (e.g., pinhole) at a specific location with respectto the user's or wearer's eye (such as the center of perspective orcenter of rotation, or elsewhere).

In some embodiments, the system may create or dynamically repositionand/or reorient one render camera for the user's left eye, and anotherrender camera for the user's right eye, as the user's eyes arephysically separated from one another and thus consistently positionedat different locations. It follows that, in at least some embodiments,virtual content rendered from the perspective of a render cameraassociated with the viewer's left eye may be presented to the userthrough an eyepiece on the left side of a head-mounted display (e.g.,head-mounted unit 602), and that virtual content rendered from theperspective of a render camera associated with the user's right eye maybe presented to the user through an eyepiece on the right side of such ahead-mounted display. Further details discussing the creation,adjustment, and use of render cameras in rendering processes areprovided in U.S. patent application Ser. No. 15/274,823, entitled“METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN3D RECONSTRUCTION,” which is expressly incorporated herein by referencein its entirety for all purposes.

In some examples, one or more modules (or components) of the system 600(e.g., light-field render controller 618, render engine 622, etc.) maydetermine the position and orientation of the render camera withinrender space based on the position and orientation of the user's headand eyes (e.g., as determined based on head pose and eye tracking data,respectively). That is, the system 600 may effectively map the positionand orientation of the user's head and eyes to particular locations andangular positions within a 3D virtual environment, place and orientrender cameras at the particular locations and angular positions withinthe 3D virtual environment, and render virtual content for the user asit would be captured by the render camera. Further details discussingreal world to virtual world mapping processes are provided in U.S.patent application Ser. No. 15/296,869, entitled “SELECTING VIRTUALOBJECTS IN A THREE-DIMENSIONAL SPACE,” which is expressly incorporatedherein by reference in its entirety for all purposes. As an example, therender controller 618 may adjust the depths at which images aredisplayed by selecting which depth plane (or depth planes) are utilizedat any given time to display the images. In some embodiments, such adepth plane switch may be carried out through an adjustment of one ormore intrinsic render camera parameters.

Registration observer 620 may use information from eye tracking module614 to identify whether the head-mounted unit 602 is properly positionedon a user's head. As an example, the eye tracking module 614 may provideeye location information, such as the positions of the centers ofrotation of the user's eyes, indicative of the three-dimensionalposition of the user's eyes relative to camera 324 and head-mounted unit602 and the eye tracking module 614 may use the location information todetermine if display 220 is properly aligned in the user's field ofview, or if the head-mounted unit 602 (or headset) has slipped or isotherwise misaligned with the user's eyes. As examples, the registrationobserver 620 may be able to determine if the head-mounted unit 602 hasslipped down the user's nose bridge, thus moving display 220 away anddown from the user's eyes (which may be undesirable), if thehead-mounted unit 602 has been moved up the user's nose bridge, thusmoving display 220 closer and up from the user's eyes, if thehead-mounted unit 602 has been shifted left or right relative the user'snose bridge, if the head-mounted unit 602 has been lifted above theuser's nose bridge, or if the head-mounted unit 602 has been moved inthese or other ways away from a desired position or range of positions.In general, registration observer 620 may be able to determine ifhead-mounted unit 602, in general, and displays 220, in particular, areproperly positioned in front of the user's eyes. In other words, theregistration observer 620 may determine if a left-eye display in displaysystem 220 is appropriately aligned with the user's left eye and aright-eye display in display system 220 is appropriately aligned withthe user's right eye. The registration observer 620 may determine if thehead-mounted unit 602 is properly positioned by determining if thehead-mounted unit 602 is positioned and oriented within a desired rangeof positions and/or orientations relative to the user's eyes.

In at least some embodiments, registration observer 620 may generateuser feedback in the form of alerts, messages, or other content. Suchfeedback may be provided to the user to inform the user of anymisalignment of the head-mounted unit 602, along with optional feedbackon how to correct the misalignment (such as a suggestion to adjust thehead-mounted unit 602 in a particular manner).

Example registration observation and feedback techniques, which may beutilized by registration observer 620, are described in U.S. patentapplication Ser. No. 15/717,747, filed Sep. 27, 2017 (Attorney DocketNo. MLEAP.052A2), which is incorporated by reference herein in itsentirety.

Example of an Eye Tracking Module

A detailed block diagram of an example eye tracking module 614 is shownin FIG. 7A. As shown in FIG. 7A, eye tracking module 614 may include avariety of different submodules, may provide a variety of differentoutputs, and may utilize a variety of available data in tracking theuser's eyes. As examples, eye tracking module 614 may utilize availabledata including eye tracking extrinsics and intrinsics, such as thegeometric arrangements of the eye-tracking camera 324 relative to thelight sources 326 and the head-mounted-unit 602; assumed eye dimensions704 such as a typical distance of approximately 4.7 mm between a user'scenter of cornea curvature and the average center of rotation of theuser's eye or typical distances between a user's center of rotation andcenter of perspective; and per-user calibration data 706 such as aparticular user's interpupillary distance. Additional examples ofextrinsics, intrinsics, and other information that may be employed bythe eye tracking module 614 are described in U.S. patent applicationSer. No. 15/497,726, filed Apr. 26, 2017 (Attorney Docket No.MLEAP.023A7), which is incorporated by reference herein in its entirety.

Image preprocessing module 710 may receive images from an eye camerasuch as eye camera 324 and may perform one or more preprocessing (i.e.,conditioning) operations on the received images. As examples, imagepreprocessing module 710 may apply a Gaussian blur to the images, maydown sample the images to a lower resolution, may applying an unsharpmask, may apply an edge sharpening algorithm, or may apply othersuitable filters that assist with the later detection, localization, andlabelling of glints, a pupil, or other features in the images from eyecamera 324. The image preprocessing module 710 may apply a low-passfilter or a morphological filter such as an open filter, which mayremove high-frequency noise such as from the pupillary boundary 516 a(see FIG. 5 ), thereby removing noise that may hinder pupil and glintdetermination. The image preprocessing module 710 may outputpreprocessed images to the pupil identification module 712 and to theglint detection and labeling module 714.

Pupil identification module 712 may receive preprocessed images from theimage preprocessing module 710 and may identify regions of those imagesthat include the user's pupil. The pupil identification module 712 may,in some embodiments, determine the coordinates of the position, orcoordinates, of the center, or centroid, of the user's pupil in the eyetracking images from camera 324. In at least some embodiments, pupilidentification module 712 may identify contours in eye tracking images(e.g., contours of pupil iris boundary), identify contour moments (i.e.,centers of mass), apply a starburst pupil detection and/or a canny edgedetection algorithm, reject outliers based on intensity values, identifysub-pixel boundary points, correct for eye-camera distortion (i.e.,distortion in images captured by eye camera 324), apply a random sampleconsensus (RANSAC) iterative algorithm to fit an ellipse to boundariesin the eye tracking images, apply a tracking filter to the images, andidentify sub-pixel image coordinates of the user's pupil centroid. Thepupil identification module 712 may output pupil identification data,which may indicate which regions of the preprocessing images module 712identified as showing the user's pupil, to glint detection and labelingmodule 714. The pupil identification module 712 may provide the 2Dcoordinates of the user's pupil (i.e., the 2D coordinates of thecentroid of the user's pupil) within each eye tracking image to glintdetection module 714. In at least some embodiments, pupil identificationmodule 712 may also provide pupil identification data of the same sortto coordinate system normalization module 718.

Pupil detection techniques, which may be utilized by pupilidentification module 712, are described in U.S. Patent Publication No.2017/0053165, published Feb. 23, 2017 and in U.S. Patent Publication No.2017/0053166, published Feb. 23, 2017, each of which is incorporated byreference herein in its entirety.

Glint detection and labeling module 714 may receive preprocessed imagesfrom module 710 and pupil identification data from module 712. Glintdetection module 714 may use this data to detect and/or identify glints(i.e., reflections off of the user's eye of the light from light sources326) within regions of the preprocessed images that show the user'spupil. As an example, the glint detection module 714 may search forbright regions within the eye tracking image, sometimes referred toherein as “blobs” or local intensity maxima, that are in the vicinity ofthe user's pupil. In at least some embodiments, the glint detectionmodule 714 may rescale (e.g., enlarge) the pupil ellipse to encompassadditional glints. The glint detection module 714 may filter glints bysize and/or by intensity. The glint detection module 714 may alsodetermine the 2D positions of each of the glints within the eye trackingimage. In at least some examples, the glint detection module 714 maydetermine the 2D positions of the glints relative to the user's pupil,which may also be referred to as the pupil-glint vectors. Glintdetection and labeling module 714 may label the glints and output thepreprocessing images with labeled glints to the 3D cornea centerestimation module 716. Glint detection and labeling module 714 may alsopass along data such as preprocessed images from module 710 and pupilidentification data from module 712.

Pupil and glint detection, as performed by modules such as modules 712and 714, may use any suitable techniques. As examples, edge detectionmay be applied to the eye image to identify glints and pupils. Edgedetection may be applied by various edge detectors, edge detectionalgorithms, or filters. For example, a Canny Edge detector may beapplied to the image to detect edges such as in lines of the image.Edges may include points located along a line that correspond to thelocal maximum derivative. For example, the pupillary boundary 516 a (seeFIG. 5 ) may be located using a Canny edge detector. With the locationof the pupil determined, various image processing techniques may be usedto detect the “pose” of the pupil 116. Determining an eye pose of an eyeimage may also be referred to as detecting an eye pose of the eye image.The pose may also be referred to as the gaze, pointing direction, or theorientation of the eye. For example, the pupil may be looking leftwardstowards an object, and the pose of the pupil could be classified as aleftwards pose. Other methods may be used to detect the location of thepupil or glints. For example, a concentric ring may be located in an eyeimage using a Canny Edge detector. As another example, anintegro-differential operator may be used to find the pupillary orlimbus boundaries of the iris. For example, the Daugmanintegro-differential operator, the Hough transform, or other irissegmentation techniques may be used to return a curve that estimates theboundary of the pupil or the iris.

3D cornea center estimation module 716 may receive preprocessed imagesincluding detected glint data and pupil identification data from modules710, 712, 714. 3D cornea center estimation module 716 may use these datato estimate the 3D position of the user's cornea. In some embodiments,the 3D cornea center estimation module 716 may estimate the 3D positionof an eye's center of cornea curvature or a user's corneal sphere, i.e.,the center of an imaginary sphere having a surface portion generallycoextensive with the user's cornea. The 3D cornea center estimationmodule 716 may provide data indicating the estimated 3D coordinates ofthe corneal sphere and/or user's cornea to the coordinate systemnormalization module 718, the optical axis determination module 722,and/or the light-field render controller 618. Further details of theoperation of the 3D cornea center estimation module 716 are providedherein in connection with FIGS. 8A-8E. Techniques for estimating thepositions of eye features such as a cornea or corneal sphere, which maybe utilized by 3D cornea center estimation module 716 and other modulesin the wearable systems of the present disclosure are discussed in U.S.patent application Ser. No. 15/497,726, filed Apr. 26, 2017 (AttorneyDocket No. MLEAP.023A7), which is incorporated by reference herein inits entirety.

Coordinate system normalization module 718 may optionally (as indicatedby its dashed outline) be included in eye tracking module 614.Coordinate system normalization module 718 may receive data indicatingthe estimated 3D coordinates of the center of the user's cornea (and/orthe center of the user's corneal sphere) from the 3D cornea centerestimation module 716 and may also receive data from other modules.Coordinate system normalization module 718 may normalize the eye cameracoordinate system, which may help to compensate for slippages of thewearable device (e.g., slippages of the head-mounted component from itsnormal resting position on the user's head, which may be identified byregistration observer 620). Coordinate system normalization module 718may rotate the coordinate system to align the z-axis (i.e., the vergencedepth axis) of the coordinate system with the cornea center (e.g., asindicated by the 3D cornea center estimation module 716) and maytranslate the camera center (i.e., the origin of the coordinate system)to a predetermined distance away from the cornea center such as 30 mm(i.e., module 718 may enlarge or shrink the eye tracking image dependingon whether the eye camera 324 was determined to be nearer or furtherthan the predetermined distance). With this normalization process, theeye tracking module 614 may be able to establish a consistentorientation and distance in the eye tracking data, relativelyindependent of variations of headset positioning on the user's head.Coordinate system normalization module 718 may provide 3D coordinates ofthe center of the cornea (and/or corneal sphere), pupil identificationdata, and preprocessed eye tracking images to the 3D pupil centerlocator module 720. Further details of the operation of the coordinatesystem normalization module 718 are provided herein in connection withFIGS. 9A-9C.

3D pupil center locator module 720 may receive data, in the normalizedor the unnormalized coordinate system, including the 3D coordinates ofthe center of the user's cornea (and/or corneal sphere), pupil locationdata, and preprocessed eye tracking images. 3D pupil center locatormodule 720 may analyze such data to determine the 3D coordinates of thecenter of the user's pupil in the normalized or unnormalized eye cameracoordinate system. The 3D pupil center locator module 720 may determinethe location of the user's pupil in three-dimensions based on the 2Dposition of the pupil centroid (as determined by module 712), the 3Dposition of the cornea center (as determined by module 716), assumed eyedimensions 704 such as the size of the a typical user's corneal sphereand the typical distance from the cornea center to the pupil center, andoptical properties of eyes such as the index of refraction of the cornea(relative to the index of refraction of air) or any combination ofthese. Further details of the operation of the 3D pupil center locatormodule 720 are provided herein in connection with FIGS. 9D-9G.Techniques for estimating the positions of eye features such as a pupil,which may be utilized by 3D pupil center locator module 720 and othermodules in the wearable systems of the present disclosure are discussedin U.S. patent application Ser. No. 15/497,726, filed Apr. 26, 2017(Attorney Docket No. MLEAP.023A7), which is incorporated by referenceherein in its entirety.

Optical axis determination module 722 may receive data from modules 716and 720 indicating the 3D coordinates of the center of the user's corneaand the user's pupil. Based on such data, the optical axis determinationmodule 722 may identify a vector from the position of the cornea center(i.e., from the center of the corneal sphere) to the center of theuser's pupil, which may define the optical axis of the user's eye.Optical axis determination module 722 may provide outputs specifying theuser's optical axis to modules 724, 728, 730, and 732, as examples.

Center of rotation (CoR) estimation module 724 may receive data frommodule 722 including parameters of the optical axis of the user's eye(i.e., data indicating the direction of the optical axis in a coordinatesystem with a known relation to the head-mounted unit 602). CoRestimation module 724 may estimate the center of rotation of a user'seye (i.e., the point around which the user's eye rotates when the usereye rotates left, right, up, and/or down). While eyes may not rotateperfectly around a singular point, assuming a singular point may besufficient. In at least some embodiments, CoR estimation module 724 mayestimate an eye's center of rotation by moving from the center of thepupil (identified by module 720) or the center of curvature of thecornea (as identified by module 716) toward the retina along the opticalaxis (identified by module 722) a particular distance. This particulardistance may be an assumed eye dimension 704. As one example, theparticular distance between the center of curvature of the cornea andthe CoR may be approximately 4.7 mm. This distance may be varied for aparticular user based on any relevant data including the user's age,sex, vision prescription, other relevant characteristics, etc.

In at least some embodiments, the CoR estimation module 724 may refineits estimate of the center of rotation of each of the user's eyes overtime. As an example, as time passes, the user will eventually rotatetheir eyes (to look somewhere else, at something closer, further, orsometime left, right, up, or down) causing a shift in the optical axisof each of their eyes. CoR estimation module 724 may then analyze two(or more) optical axes identified by module 722 and locate the 3D pointof intersection of those optical axes. The CoR estimation module 724 maythen determine the center of rotation lies at that 3D point ofintersection. Such a technique may provide for an estimate of the centerof rotation, with an accuracy that improves over time. Varioustechniques may be employed to increase the accuracy of the CoRestimation module 724 and the determined CoR positions of the left andright eyes. As an example, the CoR estimation module 724 may estimatethe CoR by finding the average point of intersection of optical axesdetermined for various different eye poses over time. As additionalexamples, module 724 may filter or average estimated CoR positions overtime, may calculate a moving average of estimated CoR positions overtime, and/or may apply a Kalman filter and known dynamics of the eyesand eye tracking system to estimate the CoR positions over time. As aspecific example, module 724 may calculate a weighted average ofdetermined points of optical axes intersection and assumed CoR positions(such as 4.7 mm from an eye's center of cornea curvature), such that thedetermined CoR may slowly drift from an assumed CoR position (i.e., 4.7mm behind an eye's center of cornea curvature) to a slightly differentlocation within the user's eye over time as eye tracking data for theuser is obtain and thereby enables per-user refinement of the CoRposition.

Interpupillary distance (IPD) estimation module 726 may receive datafrom CoR estimation module 724 indicating the estimated 3D positions ofthe centers of rotation of the user's left and right eyes. IPDestimation module 726 may then estimate a user's IPD by measuring the 3Ddistance between the centers of rotation of the user's left and righteyes. In general, the distance between the estimated CoR of the user'sleft eye and the estimated CoR of the user's right eye may be roughlyequal to the distance between the centers of a user's pupils, when theuser is looking at optical infinity (i.e., the optical axes of theuser's eyes are substantially parallel to one another), which is thetypical definition of interpupillary distance (IPD). A user's IPD may beused by various components and modules in the wearable system. Asexample, a user's IPD may be provided to registration observer 620 andused in assessing how well the wearable device is aligned with theuser's eyes (e.g., whether the left and right display lenses areproperly spaced in accordance with the user's IPD). As another example,a user's IPD may be provided to vergence depth estimation module 728 andbe used in determining a user's vergence depth. Module 726 may employvarious techniques, such as those discussed in connection with CoRestimation module 724, to increase the accuracy of the estimated IPD. Asexamples, IPD estimation module 724 may apply filtering, averaging overtime, weighted averaging including assumed IPD distances, Kalmanfilters, etc. as part of estimating a user's IPD in an accurate manner.

Interocular axis estimation module 740 may receive data from one or moremodules such as IPD estimation module 726, optical axis determinationmodule 722, 3D cornea center estimation module 716, 3D pupil centerlocator module 720, CoR estimation module 724, and CoP estimation module723. In particular, interocular axis estimation module 740 may receivedata indicating estimated 3D positions of one or more features of auser's left and right eyes, such as the left and right eye centers ofrotation, centers of perspective, pupil positions, iris positions,optical axis, interpupilary distance, etc. The interocular axisestimation module 740 may, using such information, determine a locationof an interocular axis of the user. The interocular axis may extendbetween a feature of a user's left eye and a corresponding feature ofthe user's right eye. As an example, the interocular axis may extendbetween the left-eye center of rotation and the right-eye center ofrotation. As another example, the interocular axis may extend betweenthe left-eye pupil and the right-eye pupil. The interocular axisestimation module 740 may be able to determine the position and/ororientation of the interocular axis relative to the wearable system suchas by determining the positions of the user's eyes (or some featurestherein) relative to the eye cameras 324.

Vergence depth estimation module 728 may receive data from variousmodules and submodules in the eye tracking module 614 (as shown inconnection with FIG. 7A). In particular, vergence depth estimationmodule 728 may employ data indicating estimated 3D positions of pupilcenters (e.g., as provided by module 720 described above), one or moredetermined parameters of optical axes (e.g., as provided by module 722described above), estimated 3D positions of centers of rotation (e.g.,as provided by module 724 described above), estimated IPD (e.g.,Euclidean distance(s) between estimated 3D positions of centers ofrotations) (e.g., as provided by module 726 described above), and/or oneor more determined parameters of optical and/or visual axes (e.g., asprovided by module 722 and/or module 730 described below). Vergencedepth estimation module 728 may detect or otherwise obtain a measure ofa user's vergence depth, which may be the distance from the user atwhich the user's eyes are focused. As examples, when the user is lookingat an object three feet in front of them, the user's left and right eyeshave a vergence depth of three feet; and, while when the user is lookingat a distant landscape (i.e., the optical axes of the user's eyes aresubstantially parallel to one another such that the distance between thecenters of the user's pupils may be roughly equal to the distancebetween the centers of rotation of the user's left and right eyes), theuser's left and right eyes have a vergence depth of infinity. In someembodiments, the vergence depth estimation module 728 may utilize dataindicating the estimated centers of the user's pupils (e.g., as providedby module 720) to determine the 3D distance between the estimatedcenters of the user's pupils. The vergence depth estimation module 728may obtain a measure of vergence depth by comparing such a determined 3Ddistance between pupil centers to estimated IPD (e.g., Euclideandistance(s) between estimated 3D positions of centers of rotations)(e.g., as indicated by module 726 described above). In addition to the3D distance between pupil centers and estimated IPD, the vergence depthestimation module 728 may utilize known, assumed, estimated, and/ordetermined geometries to calculate vergence depth. As an example, module728 may combine 3D distance between pupil centers, estimated IPD, and 3DCoR positions in a trigonometric calculation to estimate (i.e.,determine) a user's vergence depth. Indeed, an evaluation of such adetermined 3D distance between pupil centers against estimated IPD mayserve to indicate a measure of the user's current vergence depthrelative to optical infinity. In some examples, the vergence depthestimation module 728 may simply receive or access data indicating anestimated 3D distance between the estimated centers of the user's pupilsfor purposes of obtaining such a measure of vergence depth. In someembodiments, the vergence depth estimation module 728 may estimatevergence depth by comparing a user's left and right optical axis. Inparticular, vergence depth estimation module 728 may estimate vergencedepth by locating the distance from a user at which the user's left andright optical axes intersect (or where projections of the user's leftand right optical axes on a plane such as a horizontal plane intersect).Module 728 may utilize a user's IPD in this calculation, by setting thezero depth to be the depth at which the user's left and right opticalaxes are separated by the user's IPD. In at least some embodiments,vergence depth estimation module 728 may determine vergence depth bytriangulating eye tracking data together with known or derived spatialrelationships.

In some embodiments, vergence depth estimation module 728 may estimate auser's vergence depth based on the intersection of the user's visualaxes (instead of their optical axes), which may provide a more accurateindication of the distance at which the user is focused on. In at leastsome embodiments, eye tracking module 614 may include optical to visualaxis mapping module 730. As discussed in further detail in connectionwith FIG. 10 , a user's optical and visual axis are generally notaligned. A visual axis is the axis along which a person is looking,while an optical axis is defined by the center of that person's lens andpupil, and may go through the center of the person's retina. Inparticular, a user's visual axis is generally defined by the location ofthe user's fovea, which may be offset from the center of a user'sretina, thereby resulting in different optical and visual axis. In atleast some of these embodiments, eye tracking module 614 may includeoptical to visual axis mapping module 730. Optical to visual axismapping module 730 may correct for the differences between a user'soptical and visual axis and provide information on the user's visualaxis to other components in the wearable system, such as vergence depthestimation module 728 and light-field render controller 618. In someexamples, module 730 may use assumed eye dimensions 704 including atypical offset of approximately 5.2° inwards (nasally, towards a user'snose) between an optical axis and a visual axis. In other words, module730 may shift a user's left optical axis (nasally) rightwards by 5.2°towards the nose and a user's right optical axis (nasally) leftwards by5.2° towards the nose in order to estimate the directions of the user'sleft and right optical axes. In other examples, module 730 may utilizeper-user calibration data 706 in mapping optical axes (e.g., asindicated by module 722 described above) to visual axes. As additionalexamples, module 730 may shift a user's optical axes nasally by between4.0° and 6.5°, by between 4.5° and 6.0°, by between 5.0° and 5.4°, etc.,or any ranges formed by any of these values. In some arrangements, themodule 730 may apply a shift based at least in part upon characteristicsof a particular user such as their age, sex, vision prescription, orother relevant characteristics and/or may apply a shift based at leastin part upon a calibration process for a particular user (i.e., todetermine a particular user's optical-visual axis offset). In at leastsome embodiments, module 730 may also shift the origins of the left andright optical axes to correspond with the user's CoP (as determined bymodule 732) instead of the user's CoR.

Optional center of perspective (CoP) estimation module 732, whenprovided, may estimate the location of the user's left and right centersof perspective (CoP). A CoP may be a useful location for the wearablesystem and, in at least some embodiments, is a position just in front ofa pupil. In at least some embodiments, CoP estimation module 732 mayestimate the locations of a user's left and right centers of perspectivebased on the 3D location of a user's pupil center, the 3D location of auser's center of cornea curvature, or such suitable data or anycombination thereof. As an example, a user's CoP may be approximately5.01 mm in front of the center of cornea curvature (i.e., 5.01 mm fromthe corneal sphere center in a direction that is towards the eye'scornea and that is along the optical axis) and may be approximately 2.97mm behind the outer surface of a user's cornea, along the optical orvisual axis. A user's center of perspective may be just in front of thecenter of their pupil. As examples, a user's CoP may be less thanapproximately 2.0 mm from the user's pupil, less than approximately 1.0mm from the user's pupil, or less than approximately 0.5 mm from theuser's pupil or any ranges between any of these values. As anotherexample, the center of perspective may correspond to a location withinthe anterior chamber of the eye. As other examples, the CoP may bebetween 1.0 mm and 2.0 mm, about 1.0 mm, between 0.25 mm and 1.0 mm,between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm.

The center of perspective described herein (as a potentially desirableposition for a pinhole of a render camera and an anatomical position ina user's eye) may be a position that serves to reduce and/or eliminateundesired parallax shifts. In particular, the optical system of a user'seye is very roughly equivalent to theoretical system formed by a pinholein front of a lens, projecting onto a screen, with the pinhole, lens,and screen roughly corresponding to a user's pupil/iris, lens, andretina, respectively. Moreover, it may be desirable for there to belittle or no parallax shift when two point light sources (or objects) atdifferent distances from the user's eye are rigidly rotated about theopening of the pinhole (e.g., rotated along radii of curvature equal totheir respective distance from the opening of the pinhole). Thus, itwould seem that the CoP should be located at the center of the pupil ofan eye (and such a CoP may be used in some embodiments). However, thehuman eye includes, in addition to the lens and pinhole of the pupil, acornea that imparts additional optical power to light propagating towardthe retina). Thus, the anatomical equivalent of the pinhole in thetheoretical system described in this paragraph may be a region of theuser's eye positioned between the outer surface of the cornea of theuser's eye and the center of the pupil or iris of the user's eye. Forinstance, the anatomical equivalent of the pinhole may correspond to aregion within the anterior chamber of a user's eye. For various reasonsdiscussed herein, it may be desired to set the CoP to such a positionwithin the anterior chamber of the user's eye.

As discussed above, eye tracking module 614 may provide data, such asestimated 3D positions of left and right eye centers of rotation (CoR),vergence depth, left and right eye optical axis, 3D positions of auser's eye, 3D positions of a user's left and right centers of corneacurvature, 3D positions of a user's left and right pupil centers, 3Dpositions of a user's left and right center of perspective, a user'sIPD, etc., to other components, such as light-field render controller618 and registration observer 620, in the wearable system. Eye trackingmodule 614 may also include other submodules that detect and generatedata associated with other aspects of a user's eye. As examples, eyetracking module 614 may include a blink detection module that provides aflag or other alert whenever a user blinks and a saccade detectionmodule that provides a flag or other alert whenever a user's eyesaccades (i.e., quickly shifts focus to another point).

Example of a Render Controller

A detailed block diagram of an example light-field render controller 618is shown in FIG. 7B. As shown in FIGS. 6 and 7B, render controller 618may receive eye tracking information from eye tracking module 614 andmay provide outputs to render engine 622, which may generate images tobe displayed for viewing by a user of the wearable system. As examples,render controller 618 may receive information regarding a vergencedepth, left and right eye centers of rotation (and/or centers ofperspective), and other eye data such as blink data, saccade data, etc.

Depth plane selection module 750 may receive vergence depth informationand, based on such data, may cause render engine 622 to provide contentto a user, with the content appearing to be located on a particulardepth plane (i.e., at a particular accommodation or focal distance). Asdiscussed in connection with FIG. 4 , a wearable system may include aplurality of discrete depth planes formed by a plurality of waveguides,each conveying image information with a varying level of wavefrontcurvature. In some embodiments, a wearable system may include one ormore variable depth planes, such as an optical element that conveysimage information with a level of wavefront curvature that varies overtime. In these and other embodiments, depth plane selection module 750may cause render engine 622 to convey content to a user at a selecteddepth (i.e., cause render engine 622 to direct display 220 to switchdepth planes), based in part of the user's vergence depth. In at leastsome embodiments, depth plane selection module 750 and render engine 622may render content at different depths and also generate and/or providedepth plane selection data to display hardware such as display 220.Display hardware such as display 220 may perform an electrical depthplane switching in response to depth plane selection data (which may becontrol signals) generated by and/or provided by modules such as depthplane selection module 750 and render engine 622.

In general, it may be desirable for depth plane selection module 750 toselect a depth plane matching the user's current vergence depth, suchthat the user is provided with accurate accommodation cues. However, itmay also be desirable to switch depth planes in a discreet andunobtrusive manner. As examples, it may be desirable to avoid excessiveswitching between depth planes and/or it may be desire to switch depthplanes at a time when the user is less likely to notice the switch, suchas during a blink or eye saccade.

Hysteresis band crossing detection module 752 may help to avoidexcessive switching between depth planes, particularly when a user'svergence depth fluctuates at the midpoint or transition point betweentwo depth planes. In particular, module 752 may cause depth planeselection module 750 to exhibit hysteresis in its selection of depthplanes. As an example, modules 752 may cause depth plane selectionmodule 750 to switch from a first farther depth plane to a second closerdepth plane only after a user's vergence depth passes a first threshold.Similarly, module 752 may cause depth plane selection module 750 (whichmay in turn direct displays such as display 220) to switch to the firstfarther depth plane only after the user's vergence depth passes a secondthreshold that is farther from the user than the first threshold. In theoverlapping region between the first and second thresholds, module 750may cause depth plane selection module 750 to maintain whichever depthplane is currently select as the selected depth plane, thus avoidingexcessive switching between depth planes.

Ocular event detection module 750 may receive other eye data from theeye tracking module 614 of FIG. 7A and may cause depth plane selectionmodule 750 to delay some depth plane switches until an ocular eventoccurs. As an example, ocular event detection module 750 may cause depthplane selection module 750 to delay a planned depth plane switch until auser blink is detected; may receive data from a blink detectioncomponent in eye tracking module 614 that indicates when the user iscurrently blinking; and, in response, may cause depth plane selectionmodule 750 to execute the planned depth plane switch during the blinkevent (such by causing module 750 to direct display 220 to execute thedepth plane switch during the blink event). In at least someembodiments, the wearable system may be able to shift content onto a newdepth plane during a blink event such that the user is unlikely toperceive the shift. As another example, ocular event detection module750 may delay planned depth plane switches until an eye saccade isdetected. As discussed in connection with eye blinks, such as anarrangement may facilitate the discretely shifting of depth planes.

If desired, depth plane selection module 750 may delay planned depthplane switches only for a limited period of time before executing thedepth plane switch, even in the absence of an ocular event. Similarly,depth plane selection module 750 may execute a depth plane switch whenthe user's vergence depth is substantially outside of acurrently-selected depth plane (i.e., when the user's vergence depth hasexceeded a predetermined threshold beyond the regular threshold for adepth plane switch), even in the absence of an ocular event. Thesearrangements may help ensure that ocular event detection module 754 doesnot indefinitely delay depth plane switches and does not delay depthplane switches when a large accommodation error is present.

Render camera controller 758 may provide information to render engine622 indicating where the user's left and right eyes are. Render engine622 may then generate content by simulating cameras at the positions ofthe user's left and right eyes and generating content based on theperspectives of the simulated cameras. As discussed above, the rendercamera is a simulated camera for use in rendering virtual image contentpossibly from a database of objects in a virtual world. The objects mayhave locations and orientations relative to the user or wearer andpossibly relative to real objects in the environment surrounding theuser or wearer. The render camera may be included in a render engine torender virtual images based on the database of virtual objects to bepresented to the eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a camera (corresponding to the “rendercamera”) having an aperture, lens, and detector viewing the objects inthe virtual world. The virtual images are taken from the perspective ofsuch a camera having a position of the “render camera.” For example, thevirtual images may be rendered as if captured from the perspective of acamera having a specific location with respect to the user's or wearer'seye so as to provide images that appear to be from the perspective ofthe user or wearer. In some embodiments, the images are rendered as ifcaptured from the perspective of a camera having an aperture at aspecific location with respect to the user's or wearer's eye (such asthe center of perspective or center of rotation as discussed herein, orelsewhere).

Render camera controller 758 may determine the positions of the left andright cameras based on the left and right eye centers of rotation (CoR),determined by CoR estimation module 724, and/or based on the left andright eye centers of perspective (CoP), determined by CoP estimationmodule 732. In some embodiments, render camera controller 758 may switchbetween the CoR and CoP locations based on various factors. As examples,the render camera controller 758 may, in various modes, register therender camera to the CoR locations at all times, register the rendercamera to the CoP locations at all times, toggle or discretely switchbetween registering the render camera to the CoR locations andregistering the render camera to the CoP locations over time based onvarious factors, or dynamically register the render camera to any of arange of different positions along the optical (or visual) axis betweenthe CoR and CoP locations over time based on various factors. The CoRand CoP positions may optionally pass through smoothing filter 756 (inany of the aforementioned modes for render camera positioning) which mayaverage the CoR and CoP locations over time to reduce noise in thesepositions and prevent jitter in the render simulated render cameras.

In at least some embodiments, the render camera may be simulated as apinhole camera with the pinhole disposed at the position of theestimated CoR or CoP identified by eye tracking module 614. As the CoPis offset from the CoR, the location of the render camera and itspinhole both shift as the user's eye rotates, whenever the rendercamera's position is based on a user's CoP. In contrast, whenever therender camera's position is based on a user's CoR, the location of therender camera's pinhole does not move with eye rotations, although therender camera (which is behind the pinhole) may, in some embodiments,move with eye rotation. In other embodiments where the render camera'sposition is based on a user's CoR, the render camera may not move (i.e.,rotate) with a user's eye.

Example of a Registration Observer

A block diagram of an example registration observer 620 is shown in FIG.7C. As shown in FIGS. 6, 7A, and 7C, registration observer 620 mayreceive eye tracking information from eye tracking module 614 (FIGS. 6and 7A). As examples, registration observer 620 may receive informationon a user's left and right eye centers of rotation (e.g., thethree-dimensional positions of the user's left and right eye centers ofrotations, which may be on a common coordinate system or have a commonframe of reference with the head-mounted display system 600). As otherexamples, registration observer 620 may receive display extrinsics, fittolerances, and an eye-tracking valid indicator. The display extrinsicsmay include information on the display (e.g., display 200 of FIG. 2 )such as the field of view of the display, the size of one or moredisplay surfaces, and the positions of the display surfaces relative tothe head-mounted display system 600. The fit tolerances may includeinformation on display registration volumes, which may indicate how farthe user's left and right eyes may move from nominal positions beforedisplay performance is impacted. In addition, the fit tolerances mayindicate the amount of display performance impact that is expected as afunction of the positions of the user's eyes.

As shown in FIG. 7C, registration observer 620 may include a 3Dpositional fit module 770. The positional fit module 770 may obtain andanalyze various pieces of data including, as examples, a left eye centerof rotation 3D position (e.g., CoR Left), a right eye center of rotation3D position (e.g., CoR Right), display extrinsics, and fit tolerances.The 3D positional fit module 770 may determine how far the user's leftand right eyes are from the respective left and right eye nominalpositions (e.g., may calculate 3D left error and 3D right error) and mayprovide the error distances (e.g., 3D left error and 3D right error) todevice 3D fit module 772.

3D positional fit module 770 may also compare the error distances to thedisplay extrinsics and the fit tolerances to determine if the users eyeare within a nominal volume, a partially-degraded volume (e.g., a volumein which the performance of display 220 is partially degraded), or in afully degraded or nearly fully degraded volume (e.g., a volume in whichdisplay 220 is substantially unable to provide content to the user'seyes). In at least some embodiments, 3D positional fit module 770 or 3Dfit module 772 may provide an output qualitatively describing the fit ofthe HMD on the user, such as the Quality of Fit output shown in FIG. 7C.As an example, module 770 may provide an output indicating whether thecurrent fit of the HMD on the user is good, marginal, or failed. A goodfit may correspond to a fit that enables the user to view at least acertain percentage of the image (such as 90%), a marginal fit may enablethe user to view at least a lower percentage of the image (such as 80%),while a failed fit may be a fit in which only an even lower percentageof the image is visible to the user.

As another example, the 3D positional fit module 770 and/or device 3Dfit module 772 may calculate a visible area metric, which may be apercentage of the overall area (or pixels) of images display by display220 that are visible to the user. Modules 770 and 772 may calculate thevisible area metric by evaluating the positions of the user's left andright eyes (e.g., which may be based on the centers of rotation of theuser's eyes) relative to display 220 and using one or more models (e.g.,a mathematical or geometric model), one or more look-up tables, or othertechniques or combinations of these and other techniques to determinewhat percentage of the images are visible to the user as a function ofthe positions of the user's eyes. Additionally, modules 770 and 772 maydetermine which regions or portions of the images display by display 220are expected to be visible to the user as a function of the positions ofthe user's eyes.

Registration observer 620 may also include a device 3D fit module 772.Module 772 may receive data from 3D positional fit module 770 and mayalso receive an eye tracking valid indicator, which may be provided byeye tracking module 614 and may indicate whether the eye tracking systemis currently tracking the positions of the user's eyes or if eyetracking data is unavailable or in an error condition (e.g., determinedto be no reliable). Device 3D fit module 772 may, if desired, modifyquality of fit data received from 3D positional fit module 770 dependingon the state of the eye tracking valid data. For example, if the datafrom the eye tracking system is indicated to not be available or to havean error, the device 3D fit module 772 may provide a notification thatthere is an error and/or not provide output to the user regarding fitquality or fit errors.

In at least some embodiments, registration observer 620 may providefeedback to users on the quality of fit as well as details of the natureand magnitude of the error. As examples, the head-mounted display systemmay provide feedback to the user during calibration or fitting processes(e.g., as part of a setup procedure) and may provide feedback duringoperation (e.g., if the fit degrades due to slippage, the registrationobserver 620 may prompt the user to readjust the head-mounted displaysystem). In some embodiments, the registration analysis may be performedautomatically (e.g., during use of the head-mounted display system) andthe feedback may be provided without user input. These are merelyillustrative examples.

Example of Locating a User's Cornea with an Eye Tracking System

FIG. 8A is a schematic diagram of an eye showing the eye's cornealsphere. As shown in FIG. 8A, a user's eye 810 may have a cornea 812, apupil 822, and a lens 820. The cornea 812 may have an approximatelyspherical shape, shown by corneal sphere 814. Corneal sphere 814 mayhave a center point 816, also referred to as a corneal center, and aradius 818. The semispherical cornea of a user's eye may curve aroundthe corneal center 816.

FIGS. 8B-8E illustrate an example of locating a user's corneal center816 using 3D cornea center estimation module 716 and eye tracking module614.

As shown in FIG. 8B, 3D cornea center estimation module 716 may receivean eye tracking image 852 that includes a corneal glint 854. The 3Dcornea center estimation module 716 may then simulate, in an eye cameracoordinate system 850, the known 3D positions of the eye camera 324 andlight source 326 (which may be based on data in eye tracking extrinsics& intrinsics database 702, assumed eye dimensions database 704, and/orper-user calibration data 706) in order to cast a ray 856 in the eyecamera coordinate system. In at least some embodiments, the eye cameracoordinate system 850 may have its origin at the 3D position of theeye-tracking camera 324.

In FIG. 8C, 3D cornea center estimation module 716 simulates a cornealsphere 814 a (which may be based on assumed eye dimensions from database704) and corneal curvature center 816 a at a first position. The 3Dcornea center estimation module 716 may then check to see whether thecorneal sphere 814 a would properly reflect light from the light source326 to the glint position 854. As shown in FIG. 8C, the first positionis not a match as the ray 860 a does not intersect light source 326.

Similarly in FIG. 8D, 3D cornea center estimation module 716 simulates acorneal sphere 814 b and corneal curvature center 816 b at a secondposition. The 3D cornea center estimation module 716 then checks to seewhether the corneal sphere 814 b properly reflects light from the lightsource 326 to the glint position 854. As shown in FIG. 8D, the secondposition is also not a match.

As shown in FIG. 8E, the 3D cornea center estimation module 716eventually is able to determine the correct position of the cornealsphere is corneal sphere 814 c and corneal curvature center 816 c. The3D cornea center estimation module 716 confirms the illustrated positionis correct by checking that light from source 326 will properly reflectoff of the corneal sphere and be imaged by camera 324 at the correctlocation of glint 854 on image 852. With this arrangement and with theknown 3D positions of the light source 326, the camera 324, and theoptical properties of the camera (focal length, etc.), the 3D corneacenter estimation module 716 may determine the 3D location of thecornea's center of curvature 816 (relative to the wearable system).

The processes described herein in connection with at least FIGS. 8C-8Emay effectively be an iterative, repetitious, or optimization process toidentify the 3D position of the user's cornea center. As such, any of aplurality of techniques (e.g., iterative, optimization techniques, etc.)may be used to efficiently and quickly prune or reduce the search spaceof possible positions. Moreover, in some embodiments, the system mayinclude two, three, four, or more light sources such as light source 326and some of all of these light sources may be disposed at differentpositions, resulting in multiple glints such as glint 854 located atdifferent positions on image 852 and multiple rays such as ray 856having different origins and directions. Such embodiments may enhancethe accuracy of the 3D cornea center estimation module 716, as themodule 716 may seek to identify a cornea position that results in someor all of the glints & rays being properly reflected between theirrespective light sources and their respective positions on image 852. Inother words and in these embodiments, the positions of some or all ofthe light sources may be relied upon in the 3D cornea positiondetermination (e.g., iterative, optimization techniques, etc.) processesof FIGS. 8B-8E.

Example of Normalizing the Coordinate System of Eye Tracking Images

FIGS. 9A-9C illustrate an example normalization of the coordinate systemof eye tracking images, by a component in the wearable system such ascoordinate system normalization module 718 of FIG. 7A. Normalizing thecoordinate system of eye tracking images relative to a user's pupillocation may compensate for slippage of the wearable system relative toa user's face (i.e., headset slippage) and such normalization mayestablish a consistent orientation and distance between eye trackingimages and a user's eyes.

As shown in FIG. 9A, coordinate system normalization module 718 mayreceive estimated 3D coordinates 900 of a user's center of cornealrotation and may receive un-normalized eye tracking images such as image852. Eye tracking image 852 and coordinates 900 may be in anun-normalized coordinate system 850 that is based on the location ofeye-tracking camera 324, as an example.

As a first normalization step, coordinate system normalization module718 may rotate coordinate system 850 into rotated coordinate system 902,such that the z-axis (i.e., the vergence depth axis) of the coordinatesystem may be aligned with a vector between the origin of the coordinatesystem and cornea center of curvature coordinates 900, as shown in FIG.9B. In particular, coordinate system normalization module 718 may rotateeye tracking image 850 into rotated eye tracking image 904, until thecoordinates 900 of the user's corneal center of curvature are normal tothe plane of the rotated image 904.

As a second normalization step, coordinate system normalization module718 may translate rotated coordinate system 902 into normalizedcoordinate system 910, such that cornea center of curvature coordinates900 are a standard, normalized distance 906 from the origin ofnormalized coordinate system 910, as shown in FIG. 9C. In particular,coordinate system normalization module 718 may translate rotated eyetracking image 904 into normalized eye tracking image 912. In at leastsome embodiments, the standard, normalized distance 906 may beapproximately 30 millimeters. If desired, the second normalization stepmay be performed prior to the first normalization step.

Example of Locating a User's Pupil Centroid with an Eye Tracking System

FIGS. 9D-9G illustrate an example of locating a user's pupil center(i.e., the center of a user's pupil 822 as shown in FIG. 8A) using 3Dpupil center locator module 720 and eye tracking module 614.

As shown in FIG. 9D, 3D pupil center locator module 720 may receive anormalized eye tracking image 912 that includes a pupil centroid 913(i.e., a center of a user's pupil as identified by pupil identificationmodule 712). The 3D pupil center locator module 720 may then simulatethe normalized 3D position 910 of eye camera 324 to cast a ray 914 inthe normalized coordinate system 910, through the pupil centroid 913.

In FIG. 9E, 3D pupil center locator module 720 may simulate a cornealsphere such as corneal sphere 901 having center of curvature 900 basedon data from 3D cornea center estimation module 716 (and as discussed inmore detail in connection with FIGS. 8B-8E). As an example, the cornealsphere 901 may be positioned in the normalized coordinate system 910based on the location of the center of curvature 816 c identified inconnection with FIG. 8E and based on the normalization processes ofFIGS. 9A-9C. Additionally, 3D pupil center locator module 720 mayidentify a first intersection 916 between ray 914 (i.e., a ray betweenthe origin of normalized coordinate system 910 and the normalizedlocation of a user's pupil) and the simulated cornea, as shown in FIG.9E.

As shown in FIG. 9F, 3D pupil center locator module 720 may determinepupil sphere 918 based on corneal sphere 901. Pupil sphere 918 may sharea common center of curvature with corneal sphere 901, but have a smallradius. 3D pupil center locator module 720 may determine a distancebetween cornea center 900 and pupil sphere 918 (i.e., a radius of pupilsphere 918) based on a distance between the corneal center and the pupilcenter. In some embodiments, the distance between a pupil center and acorneal center of curvature may be determined from assumed eyedimensions 704 of FIG. 7A, from eye tracking extrinsics and intrinsicsdatabase 702, and/or from per-user calibration data 706. In otherembodiments, the distance between a pupil center and a corneal center ofcurvature may be determined from per-user calibration data 706 of FIG.7A.

As shown in FIG. 9G, 3D pupil center locator module 720 may locate the3D coordinates of a user's pupil center based on variety of inputs. Asexamples, the 3D pupil center locator module 720 may utilize the 3Dcoordinates and radius of the pupil sphere 918, the 3D coordinates ofthe intersection 916 between a simulated cornea sphere 901 and a ray 914associated with a pupil centroid 913 in a normalized eye tracking image912, information on the index of refraction of a cornea, and otherrelevant information such as the index of refraction of air (which maybe stored in eye tracking extrinsics & intrinsics database 702) todetermine the 3D coordinates of the center of a user's pupil. Inparticular, the 3D pupil center locator module 720 may, in simulation,bend ray 916 into refracted ray 922 based on refraction differencebetween air (at a first index of refraction of approximately 1.00) andcorneal material (at a second index of refraction of approximately1.38). After taking into account refraction caused by the cornea, 3Dpupil center locator module 720 may determine the 3D coordinates of thefirst intersection 920 between refracted ray 922 and pupil sphere 918.3D pupil center locator module 720 may determine that a user's pupilcenter 920 is located at approximately the first intersection 920between refracted ray 922 and pupil sphere 918. With this arrangement,the 3D pupil center locator module 720 may determine the 3D location ofthe pupil center 920 (relative to the wearable system), in thenormalized coordinate system 910. If desired, the wearable system mayun-normalize the coordinates of the pupil center 920 into the originaleye camera coordinate system 850. The pupil center 920 may be usedtogether with the corneal curvature center 900 to determine, among otherthings, a user's optical axis using optical axis determination module722 and a user's vergence depth by vergence depth estimation module 728.

Example of Differences Between Optical and Visual Axes

As discussed in connection with optical to visual mapping module 730 ofFIG. 7A, a user's optical and visual axes are generally not aligned, duein part to a user's visual axis being defined by their fovea and thatfoveae are not generally in the center of a person's retina. Thus, whena person desires to concentrate on a particular object, the personaligns their visual axis with that object to ensure that light from theobject falls on their fovea while their optical axis (defined by thecenter of their pupil and center of curvature of their cornea) isactually slightly offset from that object. FIG. 10 is an example of aneye 1000 illustrating the eye's optical axis 1002, the eye's visual axis1004, and the offset between these axes. Additionally, FIG. 10illustrates the eye's pupil center 1006, the eye's center of corneacurvature 1008, and the eye's average center of rotation (CoR) 1010. Inat least some populations, the eye's center of cornea curvature 1008 maylie approximately 4.7 mm in front, as indicated by dimension 1012, ofthe eye's average center of rotation (CoR) 1010. Additionally, the eye'scenter of perspective 1014 may lie approximately 5.01 mm in front of theeye's center of cornea curvature 1008, about 2.97 mm behind the outersurface 1016 of the user's cornea, and/or just in front of the user'spupil center 1006 (e.g., corresponding to a location within the anteriorchamber of eye 1000). As additional examples, dimension 1012 may between3.0 mm and 7.0 mm, between 4.0 and 6.0 mm, between 4.5 and 5.0 mm, orbetween 4.6 and 4.8 mm or any ranges between any values and any valuesin any of these ranges. The eye's center of perspective (CoP) 1014 maybe a useful location for the wearable system as, in at least someembodiments, registering a render camera at the CoP may help to reduceor eliminate parallax artifacts.

FIG. 10 also illustrates such a within a human eye 1000 with which thepinhole of a render camera may be aligned. As shown in FIG. 10 , thepinhole of a render camera may be registered with a location 1014 alongthe optical axis 1002 or visual axis 1004 of the human eye 1000 closerto the outer surface of the cornea than both (a) the center of the pupilor iris 1006 and (b) the center of cornea curvature 1008 of the humaneye 1000. For example, as shown in FIG. 10 , the pinhole of a rendercamera may be registered with a location 1014 along the optical axis1002 of the human eye 1000 that is about 2.97 millimeters rearward fromthe outer surface of the cornea 1016 and about 5.01 millimeters forwardfrom the center of cornea curvature 1008. The location 1014 of thepinhole of the render camera and/or the anatomical region of the humaneye 1000 to which the location 1014 corresponds may be seen asrepresenting the center of perspective of the human eye 1000. Theoptical axis 1002 of the human eye 1000 as shown in FIG. 10 representsthe most direct line through the center of cornea curvature 1008 and thecenter of the pupil or iris 1006. The visual axis 1004 of the human eye1000 differs from the optical axis 1002, as it represents a lineextending from the fovea of the human eye 1000 to the center of thepupil or iris 1006.

Example Processes of Rendering Content and Checking Registration Basedon Eye Tracking

FIG. 11 is a process flow diagram of an example method 1100 for usingeye tracking in rendering content and providing feedback on registrationin a wearable device. The method 1100 may be performed by the wearablesystem described herein. Embodiments of the method 1100 may be used bythe wearable system to render content and provide feedback onregistration (i.e., fit of the wearable device to the user) based ondata from an eye tracking system.

At block 1110, the wearable system may capture images of a user's eye oreyes. The wearable system may capture eye images using one or more eyecameras 324, as shown at least in the example of FIG. 3 . If desired,the wearable system may also include one or more light sources 326configured to shine IR light on a user's eyes and produce correspondingglints in the eye images captured by eye cameras 324. As discussedherein, the glints may be used by an eye tracking module 614 to derivevarious pieces of information about a user's eye including where the eyeis looking.

At block 1120, the wearable system may detect glints and pupils in theeye images captured in block 1110. As an example, block 1120 may includeprocessing the eye images by glint detection & labeling module 714 toidentify the two-dimensional positions of glints in the eye images andprocessing the eye images by pupil identification module 712 to identifythe two-dimensional positions of pupils in the eye images.

At block 1130, the wearable system may estimate the three-dimensionalpositions of a user's left and right corneas relative to the wearablesystem. As an example, the wearable system may estimate the positions ofthe center of curvature of a user's left and right corneas as well asthe distances between those centers of curvature and the user's left andright corneas. Block 1130 may involve 3D cornea center estimation module716 identifying the position of the centers of curvature as describedherein at least in connection with FIGS. 7A and 8A-8E.

At block 1140, the wearable system may estimate the three-dimensionalpositions of a user's left and right pupil centers relative to thewearable system. As an example, the wearable system and 3D pupil centerlocator module 720 in particular, may estimate the positions of theuser's left and right pupil centers as described at least in connectionwith FIGS. 7A and 9D-9G, as part of block 1140.

At block 1150, the wearable system may estimate the three-dimensionalpositions of a user's left and right centers or rotation (CoR) relativeto the wearable system. As an example, the wearable system and CoRestimation module 724 in particular, may estimate the positions of theCoR for the user's left and right eyes as described at least inconnection with FIGS. 7A and 10 . As a particular example, the wearablesystem may find the CoR of an eye by walking back along the optical axisfrom the center of curvature of a cornea towards the retina.

At block 1160, the wearable system may estimate a user's IPD, vergencedepth, center of perspective (CoP), optical axis, visual axis, and otherdesired attributes from eye tracking data. As examples, IPD estimationmodule 726 may estimate a user's IPD by comparing the 3D positions ofthe left and right CoRs, vergence depth estimation module 728 mayestimate a user's depth by finding an intersection (or nearintersection) of the left and right optical axes or an intersection ofthe left and right visual axes, optical axis determination module 722may identify the left and right optical axes over time, optical tovisual axis mapping module 730 may identify the left and right visualaxes over time, and the CoP estimation module 732 may identify the leftand right centers of perspective, as part of block 1160.

At block 1170, the wearable system may render content and may,optionally, provide feedback on registration (i.e., fit of the wearablesystem to the user's head) based in part on the eye tracking dataidentified in blocks 1120-1160. As an example, the wearable system mayidentify a suitable location for a render camera and then generatecontent for a user based on the render camera's location, as discussedin connection with light-field render controller 618, FIG. 7B, andrender engine 622. As another example, the wearable system may determineif it is properly fitted to the user, or has slipped from its properlocation relative to the user, and may provide optional feedback to theuser indicating whether the fit of the device needs adjustment, asdiscussed in connection with registration observer 620. In someembodiments, the wearable system may adjust rendered content based onimproper or less than ideal registration in an attempt to reduce,minimize or compensate for the effects of improper or mis-registration.

Overview of Device Registration

In order for the wearable system 200 described herein to output imagesof high perceived image quality, the display 220 of the wearable system200 (FIG. 2 ) is preferably properly fitted to a user (e.g., positionedand oriented with respect to the user's head such that the inputs andoutputs of system 200 interface appropriately with correspondingportions of the user's head and such that the device is stable andcomfortable to wear and use). As an example, for display 220 to providevisual content to a user's eyes, the display 220 is preferably situatedin front of the user's eyes and, depending on the relevant properties ofthe display 220, the user's eyes are preferably situated in a particularvolume. As additional examples, the speaker 240 is preferably situatednear, on, or in the user's ears to provide high-quality audio content tothe user, audio sensor (e.g., a microphone) 232 is preferably situatedin a particular area to receive sound from the user, and inward-facingimaging system 462 (which may include one or more cameras 324 and one ormore infrared light sources 326) is preferably properly situated in aposition and orientation to obtain clear, unobstructed images of auser's eyes (which may be part of an eye tracking system). These aremerely examples of various reasons why wearable system 200 arepreferably properly fitted to users.

In order to ensure the wearable system 200 is properly registered to auser, the wearable system 200 may include a registration observer suchas registration observer 620 of FIG. 6 . In some embodiments, theproperly registered wearable system 200 includes a display that ispositioned so that one or more eyes of the user are able to receivesufficient image light to see substantially the entirety of the field ofview provided by the display 220 of the wearable display system 200. Forexample, a properly registered display may allow an image to be seenacross about 80% or more, about 85% or more, about 90% or more, or about95% or more of the field of view of the display with a brightnessuniformity of 80% or more, about 85% or more, about 90% or more, orabout 95% or more. It will be appreciated that the brightness uniformitymay be equal to 100% times the minimum luminance divided by the maximumluminance across the entirety of the field of view of the display(100%×L_(min)/L_(max)), when the display is displaying the same contentthroughout the field of view.

The registration observer 620 may determine how the wearable system 200is fitted on the user (e.g., if the display 220 of the wearable system200 is positioned on the user properly) using various sensors. As anexample, the registration observer 620 may use an inward-facing imagingsystem 462, which may include an eye tracking system, to determine howrelevant parts of the wearable system 200 are spatially oriented withrespect to the user and, in particular, the user's eyes, ears, mouth, orother parts that interface with the wearable system 200.

The registration observer 620 may assist with a calibration process,such an initial or subsequent configuration or setup of the wearablesystem 200 for a particular user. As an example, registration observer620 may provide feedback to a user during configuration or setup of thewearable system 200 for that particular user. Additionally oralternatively, the registration observer 620 may continuously, orintermittently, monitor registration of the wearable system 200 on auser to check for continued proper registration during use and mayprovide user feedback on the fly. Registration observer 620 may provideuser feedback, either as part of a configuration process or as part ofregistration monitoring during use, that indicates when the wearablesystem 200 is properly registered and when the wearable system 200 isnot properly registered. The registration observer 620 may also provideparticular recommendations for how the user may correct anymisregistration and achieve proper registration. As examples, theregistration observer 620 may recommend the user to push the wearabledevice back up after detecting slippage of the wearable device (such asdown the user's nasal bridge), may recommend that the user adjust someadjustable component of the wearable device, etc.

Example of a Registration Coordinate System

FIGS. 12A-12B illustrate an example eye position coordinate system,which may be used for defining three-dimensional positions of a user'sleft and right eyes relative to the display of the wearable systemdescribed herein. As examples, the coordinate system may include axis x,y, and z. Axis z of the coordinate system may correspond to depth, suchthe distance between the plane a user's eyes lie in and the plane thatdisplay 220 lies in (e.g., the direction normal to the plane of thefront of a user's face). Axis x of the coordinate system may correspondto a left-right direction, such as the distance between the users leftand right eyes. In other words, axis x may generally be parallel to theinterocular axis (e.g., a line between matching left and right eyefeatures, such as centers of rotation). Axis y of the coordinate systemmay correspond to an up-down direction, which may be a verticaldirection when the user is upright.

FIG. 12A illustrates a side view of a user's eye 1200 and a displaysurface 1202 (which may be a part of display 220 of FIG. 2 ), while FIG.12B illustrates a top down view of the user's eye 1200 and the displaysurface 1202. Display surface 1202 may be located in front of the user'seyes and may output image light to the user's eyes. As an example,display surface 1202 may comprise one or more out-coupling lightelements, active or pixel display elements, and may be part of a stackof waveguides, such as stacked waveguide assembly 480 of FIG. 4 . Insome embodiments, the display surface 1202 may be planar. In some otherembodiments, the display surface 1202 may have other topologies (e.g.,be curved). It will be appreciated that the display surface 1202 may bea physical surface of the display, or simply a plane or other imaginarysurface from which image light is understood to propagate from thedisplay 220 to the user's eyes.

As shown in FIG. 12A, the user's eye 1200 may have an actual position1204 offset from a nominal position 1206 and the display surface 1202may be at position 1214. FIG. 12A also illustrates the corneal apex 1212of the user's eye 1200. The user's line of sight (e.g., their opticaland/or visual axis) may be substantially along the line between theactual position 1204 and the corneal apex 1212. As shown in FIGS. 12Aand 12B, the actual position 1204 may be offset from the nominalposition 1206 by an z-offset 1210, a y-offset 1208, and an x-offset1209. The nominal position 1206 may represent a preferred position(sometimes referred to as a design position, which may be generallycentered within a desired volume) for the user's eye 1200 with respectto the display surface 1202. As the user's eye 1200 moves away from thenominal position 1206, the performance of display surface 1202 may bedegraded.

It will be appreciated that a point or volume associated with the user'seye 1200 may be used to represent the position of the user's eye inanalyses of registration herein. The representative point or volume maybe any point or volume associated with the eye 1200, and preferably isconsistently used. For example, the point or volume may be on or in theeye 1200, or may be disposed away from the eye 1200. In someembodiments, the point or volume is the center of rotation of the eye1200. The center of rotation may be determined as described herein andmay have advantages for simplifying the registration analyses, since itis roughly symmetrically disposed on the various axes within the eye1200 and allows a single display registration volume aligned with theoptical axis to be utilized for the analyses.

FIG. 12A also illustrates that the display surface 1202 may be centeredbelow the user's horizon (as seen along the y-axis when the user islooking straight ahead, with their optical axis parallel to the ground)and may be tilted (with respect to the y-axis). In particular, thedisplay surface 1202 may be disposed somewhat below the user's horizonsuch that the user would have to look downward, at approximately theangle 1216, to look at the center of the display surface 1202, when theeye 1200 is at position 1206. This may facilitate a more natural andcomfortable interaction with the display surface 1202, particularly whenviewing content rendered at shorter depths (or distances from the user),as users may be more comfortable viewing content below their horizonthan above their horizon. Additionally, display surface 1202 may betilted, such as at angle 1218 (with respect to the y-axis) such that,when the user is looking at the center of the display surface 1202(e.g., looking slightly below the user's horizon), the display surface1202 is generally perpendicular to the user's line of sight. In at leastsome embodiments, the display surface 1202 may also be shifted left orright (e.g., along the x-axis) relative to the nominal position of theuser's eye. As an example, a left-eye display surface may be shiftedright-wards and a right-eye display surface may be shifted left-wards(e.g., display surfaces 1202 may be shifted towards each other) suchthat the user's lines of sight hits the centers of the display surfaceswhen focused at some distance less than infinity, which may increaseuser comfort during typical usage on the wearable device.

Example of Vertically Aligning Left and Right-Eye Displays on a User'sHead

As previously discussed, proper alignment between left-eye and right-eyedisplays of a head-mounted display (HMD) (e.g., display 220 in wearablesystem 200 of FIG. 3 ) and a user's eyes may be important in order toensure a desirable user experience. For example, when a left-eye displayis vertically offset, such as from normal wear and tear of the deviceover time or by other deformation means, from the user's left eye by adifferent amount than a right-eye display is vertically offset from theuser's right eye, the user may experience discomfort and eye strain.

In some embodiments, a head-mounted display system may be configured todetermine whether its left-eye and right-eye displays are level with theuser's eyes (e.g., their interocular axis) and provide appropriate userfeedback encouraging the user to adjust an image displayed on a left orright eyepiece, with the ultimate goal of correcting a calibrationmetric of a left or right display sub system (such as a projector inputfor a corresponding left or right eyepiece) so that images projectedthrough left-eye and right-eye displays are level with an interocularaxis between the eyes.

The display system may then provide the user with left eye and right eyeimages having alignment markers. The alignment markers may be providedsuch that any vertical offset between a left-eye display and a right-eyedisplay is visible to the user. In other words, if there is a left-eyedisplay and right-eye display vertical offset (e.g., the left eyedisplay is vertically offset relative to the right eye display), theuser may perceive the images from the left-eye display and right-eye asvertically misaligned and be unable to properly fuse the image (e.g.,the offset may be visible in the form of horizontal portions of thealignment markers not lining up with each other). It will be appreciatedthat the horizontal portions of the alignment markers are may take theform of other mirror image shapes or arrangements of lines which do notcompletely overlap when vertically misaligned.

FIGS. 13 and 14 illustrate example display screens that may be providedto a user of a head-mounted display (HMD) (e.g., display 220, FIG. 3 )as part of leveling the HMD (e.g., leveling a horizontal axis extendingthrough the left-eye and right-eye displays of the HMD) with respect tothe user's eyes (e.g., their interocular axis). In particular, FIG. 13illustrates a display screen 1300 that includes example guidance thatmay be provided to assist the user in leveling the HMD.

The display screen 1300 may also include feedback on the currentorientation of the HMD relative to the user's eyes. Such feedback maytake the form of an avatar 1304 (e.g., a virtual representation of auser) and a virtual depiction 1302 of the HMD. The avatar 1304 may bedepicted in a vertical orientation (such that the interocular axis is ina horizontal orientation), while the orientation of the virtualdepiction 1302 of the HMD may be tilted with respect to the horizontal.The amount and direction of tilt of the virtual depiction 1302 may bebased the estimated orientation of the interocular axis (e.g., asdetermined by module 740 of FIG. 7A) relative to the HMD. In otherembodiments, the avatar 1302 and 1304 may be omitted or may be static.In various embodiments, the amount and direction of the tilt of the HMDmay be conveyed to users via other indicia such as alignment markers1306 and 1308. Alignment marker 1306 may correspond to the user'sinterocular axis (e.g., be presented as a horizontal line), whilealignment marker 1308 may be indicative of the tilt of the HMD relativeto the interocular axis. In one example, mark 1306 may remain staticwhile mark 1308 shifts as the user adjusts the HMD on their head towardslevel. In some embodiments, the orientation of the HMD may be determinedusing various sensors attached to the HMD, including gyroscopes,accelerometers, and/or imaging systems which image one or more referencepoints in the external environment and determine the orientation of theHMD based on those reference points. In general, the alignment markersmay take any suitable shape or form. Using the alignment markers, a usermay be able to quickly perceive if the HMD is tilted on their head,including in which direction and by how much. Thus, the user can correctthe tilt of the HMD until the HMD is properly leveled.

As shown in FIG. 14 , when the user successfully adjusts the tilt of theHMD such that the HMD is level on their head (e.g., when the interocularaxis or a portion thereof is parallel to a horizontal axis extendingbetween the left and right eye displays of the HMD, when the interocularaxis or a portion thereof falls within a predetermined volume definedrelative to the HMD, etc.), the user may be presented with a displayscreen, such as screen 1400, that indicates success in leveling in theHMD. In the example of FIG. 14 , a user has successfully leveled the HMDand is shown marker 1402. Marker 1402 may appear when marks 1306 and1308 of FIG. 13 overlap, as an example.

Following leveling of the HMD on the user, the system may perform aleft-eye and right-eye display alignment protocol to properly level thedisplay of images through rigid displays that do not themselvesvertically align. In particular, the display alignment process may checkfor and identify (e.g., automatically or based on user feedback) anydifference in vertical alignment between the left and right displays andtheir respective eyes. In other words, the display alignment process maycheck (e.g., automatically or by providing a user with alignment markersthat highlight any misalignment to the user and receiving user feedbackon any misalignment) if the user's left eye has a first verticalalignment with the HAMID's left-eye display (such as a first offsetalong the Y-axis of FIG. 12A) while the user's right eye has a secondand different vertical alignment with the HMD's right-eye display (suchas a second offset, of different magnitude, along the Y-axis of FIG. 12). It will be appreciated that differences in vertical alignment betweenthe left and right sides may occur even when the HMD is worn level tothe user's head, due to various factors such warping or damage to theHMD which may cause one or both of the left-eye and right-eye displaysto shift such that they are not at the same vertical height even whenthe HMD is determined to be level with the interocular axis of theuser's eyes.

FIGS. 15 and 16 illustrate example display screens that may be providedto a user of a head-mounted display (HMD) (e.g., display 220, FIG. 3 )as part of identifying vertical misalignment remaining after leveling.As shown in FIG. 15 , a display screen such as screen 1500 may includeuser instructions as well as alignment markers such as markers 1502,1504, and 1506 that highlight any vertical misalignment to the user. Insome embodiments, the alignment markers make take the form of the letter“T” laying sideways. It will be appreciated, however, that the alignmentmarkers may take other shapes or forms.

As shown in FIG. 16 , the HMD may provide different respective alignmentmarkers to the user's left and right eyes in order to demonstrate anyleft-right vertical misalignment. For example, the HMD may displayscreen 1600 a to a user's left eye and may display screen 1600 b to theuser's right eye. Screen 1600 a may include a left-eye horizontalalignment marker 1502 and a vertical alignment marker 1506, while screen1600 b may include a right-eye horizontal alignment marker 1504 and avertical alignment marker 1506. In some embodiments, screen 1600 a maynot include the right-eye horizontal alignment marker 1504, and screen1600 b may not include the left-eye horizontal alignment marker 1502.

When a user wearing the HMD views the screens 1600 a and 1600 bsimultaneously (e.g., with their left and right eyes respectively), thevertical alignment markers 1506 a and 1506 b may appear to the user asbeing fused together (e.g., as mark 1506). However, at least because theleft-eye and right-eye horizontal alignment markers 1502 and 1504 arenot spatially aligned to one another, the user does not perceive themarks 1502 and 1504 as being fused together. Instead, the user will beable to determine if the marks 1502 and 1504 are aligned with each othervertically (or if there is a left-right vertical misalignment). In otherwords, an incorrectly aligned system will display the intended imageryof 1604 as the exemplary misaligned imagery 1602.

In at least some embodiments, alignment markers 1502 and 1504 areinitially presented at the same positions of the left and right eyedisplays, though deformation may prompt perception of the positions asnon-aligned. In some embodiments, the system may intentionally introducea vertical offset between alignment markers 1502 and 1504. In suchembodiments, the alignment of markers 1502 and 1504 may not necessarilybe representative of the alignment of the left and right eye displays.For instance, although the left and right eye displays may exhibitrelatively little or no misalignment, in these embodiments, the markers1502 and 1504 presented by the system may exhibit a relatively highamount of misalignment. Doing so may serve to promote user engagement inthe display alignment process. That is, some users may feel morecompelled to actively participate in the display alignment process whenthe markers 1502 and 1504, as presented by the system, exhibit arelatively high degree of misalignment. In some examples, the verticalpositions of the alignment markers 1502 and 1504 may be randomized,pseudorandomized, or quasi-randomized. For example, the system mayrandomly, pseudorandomly, or quasi-randomly select the verticalpositions of the alignment markers 1502 and 1504 from a predeterminedrange of vertical positions. In some embodiments, the system mayrandomly, pseudorandomly, or quasi-randomly select the offset betweenthe alignment markers 1502 and 1504 from a predetermined range ofoffsets. In some embodiments, the display system may be configured torandomly, pseudorandomly, or quasi-randomly select the offset based onpseudorandom number generators, such as a Mersenne Twister, Xorshift,and so on. In some embodiments, the pseudorandom number generators maygenerate the offset values directly. In some other embodiments, eachoffset may be associated with a number, or set of numbers, and thegeneration of a number by a pseudorandom number generators may be usedto select the offset associated with that generated number or with theset to which that generated number belongs (e.g., each offset may beassociated with one or more unique numbers or sets of numbers, a numbermay be generated using the pseudorandom generator, and the offset to beapplied may be selected based on the correspondence between thatgenerated number (or the set to which the number belongs) and thenumbers (or sets of numbers) associated with the offsets).

The system may take the selected vertical positions of the markers 1502and 1504 and/or the selected offset between the markers 1502 and 1504into account when the user provides input to vertically align themarkers 1502 and 1504. After the user provides input to vertically alignthe marks 1502 and 1504, the system may be able to determine themagnitude and direction of the left-right vertical misalignment based onthe magnitude and direction of the user input. Preferably, adjustmentsto the vertical alignment is performed on only one display at a time.

For embodiments in which the HMD includes a stack of waveguides,adjustments to vertical alignment may be performed on only one waveguideat a time. As such, in some embodiments, the system may conduct adisplay alignment process for each waveguide included in the waveguidestack of the HMD. In some examples, the waveguide stack of the HMD mayinclude a plurality of waveguides, e.g., three waveguides, for eachdepth plane. The number of waveguides may correspond to the number ofcomponent colors used by the display system to form full-color content.For instance, for RGB (red, green, blue) display systems, the threewaveguides provided for each depth plane may include a waveguideconfigured to output red light, a waveguide configured to output greenlight, and a waveguide configured to output blue light. The system maypresent markers 1502 and 1504 each component color, e.g., in red whenconducting display alignment for a waveguide configured to output redlight, in green when conducting display alignment for a waveguideconfigured to output green light, and in blue when conducting displayalignment for a waveguide configured to output blue light. In someembodiments, the system may conduct display alignment for each waveguidein succession (e.g., the alignment may be conducted sequentially fordifferent waveguides corresponding to different component colors and/orfor different waveguides providing different amounts of wavefrontdivergence corresponding to different depth planes). In some otherembodiments, the system may conduct display alignment for some or all ofthe waveguides in the waveguide stack of the HMD contemporaneously. Insuch embodiments, markers 1502 and 1504 may be presented in variouscolors (e.g., red, green, and blue versions of markers 1502 and 1504)and/or at various depths (with waveguides providing different amounts ofwavefront divergence) simultaneously, and the user may be able toprovide input to vertically shift one marker at a time.

It will be appreciated that the user input may be provided using anyinput device available to the user. For example, gestures, voiceactivated commands, physical toggles, etc. may be utilized to providethe input. As a particular example, FIG. 17 is a perspective view of auser's hand 1700 and a controller 1701 including various input devices1702, 1704, 1706. Any of the input devices 1702, 1704, 1706 may beconfigured to provide inputs regarding the vertical shift desired toalign the alignment markers 1502 and 1504.

Example Processes of Vertically Aligning Left and Right-Eye Displays ona User's Head

FIG. 18 is a process flow diagram of an example method 1800 for levelinga head-mounted display system on a user's head and for adjusting avertical alignment of left and right displays of the display system. Themethod 1800 may be performed by the wearable system described herein.

At block 1802, the wearable system may determine a potential need toadjust a calibration of the head-mounted display (HMD) (e.g., display220, FIG. 3 ). The wearable system may determine there is a need tocalibrate the HMD based on a variety of factors, individually or incombination, including a number of hours since the last calibration, adetection of potential or actual deformation of the HMD (e.g., detectionof an impact event with an accelerometer or other sensor), detection ofa misregistration of the HMD (e.g., a determination by eye trackingcomponents that one or both of the user's eyes are not in the designedposition relative to the HMD), upon start-up of the HMD, upon launchingan application on the HMD, upon determining that the HMD has been placedon the head of a user, etc. As additional options, the calibrationprocess may be recommended to users by a user manual, customer service,or other source and block 1802 may involve receiving a user request toperform the calibration process.

At block 1804, the wearable system may perform a device leveling flow,which may include blocks such as blocks 1806 and 1808. At block 1806,the HMD may determine the interocular axis of the user and the device'shorizontal axis, which may be an interdisplay axis that runs between theHMD's left-eye and right-eye displays, or measured relative to a gravityvector of a world coordinate frame of the HMD. Block 1806 may involvedetermining if the interocular axis and the device's horizontal axis areparallel, or if there is a tilt between the two axes. In some examples,block 1806 may involve determining whether the interocular axis or aportion thereof falls within a predetermined volume defined relative tothe HMD. At block 1808, the HMD may provide feedback on leveling the HMDto the user. Such feedback may take the form of displaying the alignmentmarkers 1306 and 1308 of FIG. 13 . Blocks 1806 and 1808 may continueuntil the user exits the alignment process or successfully levels theHMD on their head (e.g., aligns mark 1308 with mark 1306).

At block 1810, the wearable system may perform a display alignment flow,which may include blocks such as block 1812 and 1814. At block 1812, theHMD may provide unfused left-eye and right-eye alignment markers. As anexample, the HMD may display alignment marker 1502 of FIGS. 15 and 16 ona left-eye display and may display alignment marker 1504 of FIGS. 15 and16 on a right-eye display, as discussed herein. As described above, insome examples, the HMD may display one or both of alignment markers 1502and 1504 at randomly-, pseudorandomly-, or quasi-randomly-selectedvertical positions. At block 1814, the wearable system may receive userfeedback on left-eye or right-eye display vertical alignmentadjustments. In particular and as discussed in connection with FIGS.15-17 , the wearable system may receive user inputs that shift at leastone of the alignment markers 1502 and 1504 until the markers arevertically aligned with each other from the perspective of the user.Blocks 1812 and 1814 may continue until the user exits the alignmentprocess or accepts any vertical adjustments they have made to thealignment markers. As described above, in some examples, the wearablesystem may conduct one display alignment process per waveguide includedin the HMD. In these examples, the wearable system may perform theoperations associated with one or more of blocks 1810, 1812, and 1814for each waveguide included in the HMD.

At block 1816, the wearable system may implement any adjustmentsprovided by the user to the left-right vertical alignment. As anexample, if the user feedback indicated that the right-eye display waslower than the left-eye display by a given amount, the wearable systemmay compensate for that misalignment by shifting image content displayed(e.g., displayed following completion of the alignment process of FIG.18 ) via the right-eye display upwards by the given amount, by shiftingimage content displayed via the left-eye display downwards by the givenamount, or by an appropriate combination of shifts of the right and lefteye content.

In some embodiments, at block 1816, the wearable system may makeadjustments to one or more extrinsic parameters of a render cameraassociated with the left-eye display and to one or more extrinsicparameters of a render camera associated with the right-eye display. Forexample, in these embodiments, the wearable system may adjust theposition and/or orientation of one or both of two render camerasassociated with the two displays (e.g., the left-eye display and theright-eye display) based at least in part on the input provided by theuser. In some embodiments, the wearable system may skip the operationsassociated with blocks 1804, 1806, and/or 1808. In these embodiments,the wearable system may proceed to block 1810 immediately afterperforming the operations associated with block 1802.

Computer Vision to Detect Objects in Ambient Environment

As discussed above, the display system may be configured to detectobjects in or properties of the environment surrounding the user. Thedetection may be accomplished using a variety of techniques, includingvarious environmental sensors (e.g., cameras, audio sensors, temperaturesensors, etc.), as discussed herein.

In some embodiments, objects present in the environment may be detectedusing computer vision techniques. For example, as disclosed herein, thedisplay system's forward-facing camera may be configured to image theambient environment and the display system may be configured to performimage analysis on the images to determine the presence of objects in theambient environment. The display system may analyze the images acquiredby the outward-facing imaging system to perform scene reconstruction,event detection, video tracking, object recognition, object poseestimation, learning, indexing, motion estimation, or image restoration,etc. As other examples, the display system may be configured to performface and/or eye recognition to determine the presence and location offaces and/or human eyes in the user's field of view. One or morecomputer vision algorithms may be used to perform these tasks.Non-limiting examples of computer vision algorithms include:Scale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), binary robust invariantscalable keypoints (BRISK), fast retina keypoint (FREAK), Viola-Jonesalgorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunkalgorithm, Mean-shift algorithm, visual simultaneous location andmapping (vSLAM) techniques, a sequential Bayesian estimator (e.g.,Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

One or more of these computer vision techniques may also be usedtogether with data acquired from other environmental sensors (such as,e.g., microphone) to detect and determine various properties of theobjects detected by the sensors.

As discussed herein, the objects in the ambient environment may bedetected based on one or more criteria. When the display system detectsthe presence or absence of the criteria in the ambient environment usinga computer vision algorithm or using data received from one or moresensor assemblies (which may or may not be part of the display system),the display system may then signal the presence of the object.

Machine Learning

A variety of machine learning algorithms may be used to learn toidentify the presence of objects in the ambient environment. Oncetrained, the machine learning algorithms may be stored by the displaysystem. Some examples of machine learning algorithms may includesupervised or non-supervised machine learning algorithms, includingregression algorithms (such as, for example, Ordinary Least SquaresRegression), instance-based algorithms (such as, for example, LearningVector Quantization), decision tree algorithms (such as, for example,classification and regression trees), Bayesian algorithms (such as, forexample, Naive Bayes), clustering algorithms (such as, for example,k-means clustering), association rule learning algorithms (such as, forexample, a-priori algorithms), artificial neural network algorithms(such as, for example, Perceptron), deep learning algorithms (such as,for example, Deep Boltzmann Machine, or deep neural network),dimensionality reduction algorithms (such as, for example, PrincipalComponent Analysis), ensemble algorithms (such as, for example, StackedGeneralization), and/or other machine learning algorithms. In someembodiments, individual models may be customized for individual datasets. For example, the wearable device may generate or store a basemodel. The base model may be used as a starting point to generateadditional models specific to a data type (e.g., a particular user), adata set (e.g., a set of additional images obtained), conditionalsituations, or other variations. In some embodiments, the display systemmay be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

The criteria for detecting an object may include one or more thresholdconditions. If the analysis of the data acquired by the environmentalsensor indicates that a threshold condition is passed, the displaysystem may provide a signal indicating the detection the presence of theobject in the ambient environment. The threshold condition may involve aquantitative and/or qualitative measure. For example, the thresholdcondition may include a score or a percentage associated with thelikelihood of the reflection and/or object being present in theenvironment. The display system may compare the score calculated fromthe environmental sensor's data with the threshold score. If the scoreis higher than the threshold level, the display system may detect thepresence of the reflection and/or object. In some other embodiments, thedisplay system may signal the presence of the object in the environmentif the score is lower than the threshold. In some embodiments, thethreshold condition may be determined based on the user's emotionalstate and/or the user's interactions with the ambient environment.

In some embodiments, the threshold conditions, the machine learningalgorithms, or the computer vision algorithms may be specialized for aspecific context. For example, in a diagnostic context, the computervision algorithm may be specialized to detect certain responses to thestimulus. As another example, the display system may execute facialrecognition algorithms and/or event tracing algorithms to sense theuser's reaction to a stimulus, as discussed herein.

It will be appreciated that each of the processes, methods, andalgorithms described herein and/or depicted in the figures may beembodied in, and fully or partially automated by, code modules executedby one or more physical computing systems, hardware computer processors,application-specific circuitry, and/or electronic hardware configured toexecute specific and particular computer instructions. For example,computing systems may include general purpose computers (e.g., servers)programmed with specific computer instructions or special purposecomputers, special purpose circuitry, and so forth. A code module may becompiled and linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted programming language.In some embodiments, particular operations and methods may be performedby circuitry that is specific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (140), the remote processingmodule (150), and remote data repository (160). The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

Other Considerations

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems mayinclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someembodiments, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products. Many embodimentvariations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the embodiments described in thisdisclosure may be readily apparent to those skilled in the art, and thegeneric principles defined herein may be applied to other embodimentswithout departing from the spirit or scope of this disclosure. Thus, theclaims are not intended to be limited to the embodiments shown herein,but are to be accorded the widest scope consistent with this disclosure,the principles and the novel features disclosed herein.

Certain features that are described in this specification in the contextof separate embodiments also may be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment also may be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination may in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted may be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations may be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other embodiments. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system components in the embodiments describedabove should not be understood as requiring such separation in allembodiments, and it should be understood that the described programcomponents and systems may generally be integrated together in a singlesoftware product or packaged into multiple software products.Additionally, other embodiments are within the scope of the followingclaims. In some cases, the actions recited in the claims may beperformed in a different order and still achieve desirable results.

What is claimed is:
 1. An augmented reality system comprising: ahead-mounted display configured to present virtual content by outputtinglight to a user, the head-mounted display comprising a left-eye displayincluding a first waveguide stack comprising a plurality of waveguidesconfigured to present virtual content to the user's left eye and aright-eye display including a second waveguide stack comprising aplurality of waveguides configured to present virtual content to theuser's right eye; and at least one processor communicatively coupled tothe head-mounted display, the at least one processor configured to:conduct a plurality of alignment processes for the first waveguide stackand the second waveguide stack, wherein an alignment process isconducted for each waveguide of the first and second waveguide stacks,wherein each alignment process comprises: providing, with a waveguide ofthe left-eye display being aligned, a left-eye alignment marker;providing, with a waveguide of the right-eye display being aligned, aright-eye alignment marker; receiving user input to adjust at least oneof the left-eye and the right-eye alignment markers; and verticallyadjusting image content in the waveguide being aligned of at least oneof the left-eye and right-eye displays based on the received user input.2. The augmented reality system of claim 1, wherein the left-eyealignment marker comprises a first horizontal line and wherein theright-eye alignment marker comprises a second horizontal line.
 3. Theaugmented reality system of claim 2, wherein the processor is configuredto receive user input to adjust at least one of the left-eye andright-eye alignment markers in the form of user input to raise or lowerat least one of the first and second horizontal lines.
 4. The augmentedreality system of claim 2, wherein the processor is configured toreceive user input to adjust at least one of the left-eye and right-eyealignment markers in the form of user input to raise or lower at leastone of the first and second horizontal lines until the first and secondhorizontal lines are level from the perspective of the user.
 5. Theaugmented reality system of claim 1, wherein the first waveguide stackis configured to pass light from the world into the left eye of the userand the second waveguide stack is configured to pass light from theworld into the right eye of the user.
 6. The augmented reality system ofclaim 1, wherein the first waveguide stack is configured to pass lightfrom the world into the left eye of the user and the second waveguidestack is configured to pass light from the world into the right eye ofthe user, wherein one or more waveguides of each waveguide stack areconfigured to output light to the user with a different amount ofwavefront divergence than one or more other waveguides of that waveguidestack, wherein different amounts of wavefront divergence are associatedwith different accommodation by the eye, and wherein the outputted lightwith different amounts of wavefront divergence forms virtual objects atdifferent perceived depths away from the user.
 7. The augmented realitysystem of claim 1, wherein to provide, with the left-eye display and theright-eye display, the left-eye alignment marker and the right-eyealignment marker, respectively, the at least one processor is configuredto: provide, with the left-eye display, a first vertical alignmentmarker and a first horizontal alignment marker; and provide, with theright-eye display, a second vertical alignment marker and a secondhorizontal alignment marker.
 8. The augmented reality system of claim 7,wherein, when viewed by the user, the first and second verticalalignment markers fuse together in the user's vision and the first andsecond horizontal alignment markers remain unfused in the user's vision.9. The augmented reality system of claim 7, wherein to receive userinput to adjust at least one of the left-eye and the right-eye alignmentmarkers, the at least one processor is configured to: receive user inputto move at least one of the first and second horizontal alignmentmarkers vertically.
 10. The augmented reality system of claim 7, whereinto receive user input to adjust at least one of the left-eye and theright-eye alignment markers, the at least one processor is configuredto: receive user input to move at least one of the first and secondhorizontal alignment markers vertically until the first and secondhorizontal alignment markers are vertically aligned with each other inthe user's vision.
 11. The augmented reality system of claim 1, whereinthe at least one processor is further configured to: select a firstvertical position at which to present the left-eye alignment marker; andselect a second vertical position at which to present the right-eyealignment marker, and wherein to provide, with the left-eye display andthe right-eye display, the left-eye alignment marker and the right-eyealignment marker, respectively, the at least one processor is configuredto: provide, with the left-eye display, the left-eye alignment marker atthe first vertical position; and provide, with the right-eye display,the right-eye alignment marker at the second vertical position.
 12. Theaugmented reality system of claim 11, wherein to select the firstvertical position at which to present the left-eye alignment marker andselect the second vertical position at which to present the right-eyealignment marker, the at least one processor is configured to: pseudo-or quasi-randomly select the first vertical position at which to presentthe left-eye alignment marker; and pseudo- or quasi-randomly select thesecond vertical position at which to present the right-eye alignmentmarker.
 13. The augmented reality system of claim 11, wherein tovertically adjust image content in the at least one of the left-eye andright-eye displays based on the received user input, the at least oneprocessor is configured to: vertically adjust image content in the atleast one of the left-eye and right-eye displays based on the receiveduser input, the first vertical position selected, and the secondvertical position selected.
 14. The augmented reality system of claim 1,wherein to vertically adjust image content in the at least one of theleft-eye and right-eye displays based on the received user input, the atleast one processor is configured to: adjust one or more extrinsicparameters of at least one of a first render camera that is associatedwith the left-eye display and a second render camera that is associatedwith the right-eye display.
 15. The augmented reality system of claim14, wherein the one or more extrinsic parameters include at least one ofa position and an orientation.
 16. The augmented reality system of claim1, further comprising an imaging system configured to capture images ofeyes of the user, and wherein the at least one processor iscommunicatively coupled to the imaging system, the at least oneprocessor further configured to: determine whether the head-mounteddisplay is level relative to the user's left and right eyes based atleast in part on one or more images captured by the imaging system. 17.The augmented reality system of claim 16, wherein the at least oneprocessor is configured to provide, with the left-eye display and theright-eye display, the left-eye alignment marker and the right-eyealignment marker, respectively, in response to a determination that thehead-mounted display is level relative to the user's left and righteyes.